263-3010-00L Big Data
| Semester | Autumn Semester 2018 |
| Lecturers | G. Fourny |
| Periodicity | yearly recurring course |
| Language of instruction | English |
Courses
| Number | Title | Hours | Lecturers | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 263-3010-00 V | Big Data | 3 hrs |
| G. Fourny | ||||||||||||||||||
| 263-3010-00 U | Big Data | 2 hrs |
| G. Fourny | ||||||||||||||||||
| 263-3010-00 A | Big Data Individual work to get hands-on experience with the technologies covered, no fixed presence required. | 2 hrs | G. Fourny |
Catalogue data
| Abstract | The key challenge of the information society is to turn data into information, information into knowledge, knowledge into value. This has become increasingly complex. Data comes in larger volumes, diverse shapes, from different sources. Data is more heterogeneous and less structured than forty years ago. Nevertheless, it still needs to be processed fast, with support for complex operations. |
| Learning objective | This combination of requirements, together with the technologies that have emerged in order to address them, is typically referred to as "Big Data." This revolution has led to a completely new way to do business, e.g., develop new products and business models, but also to do science -- which is sometimes referred to as data-driven science or the "fourth paradigm". Unfortunately, the quantity of data produced and available -- now in the Zettabyte range (that's 21 zeros) per year -- keeps growing faster than our ability to process it. Hence, new architectures and approaches for processing it were and are still needed. Harnessing them must involve a deep understanding of data not only in the large, but also in the small. The field of databases evolves at a fast pace. In order to be prepared, to the extent possible, to the (r)evolutions that will take place in the next few decades, the emphasis of the lecture will be on the paradigms and core design ideas, while today's technologies will serve as supporting illustrations thereof. After visiting this lecture, you should have gained an overview and understanding of the Big Data landscape, which is the basis on which one can make informed decisions, i.e., pick and orchestrate the relevant technologies together for addressing each business use case efficiently and consistently. |
| Content | This course gives an overview of database technologies and of the most important database design principles that lay the foundations of the Big Data universe. The material is organized along three axes: data in the large, data in the small, data in the very small. A broad range of aspects is covered with a focus on how they fit all together in the big picture of the Big Data ecosystem. - physical storage: distributed file systems (HDFS), object storage(S3), key-value stores - logical storage: document stores (MongoDB), column stores (HBase), graph databases (neo4j), data warehouses (ROLAP) - data formats and syntaxes (XML, JSON, RDF, Turtle, CSV, XBRL, YAML, protocol buffers, Avro) - data shapes and models (tables, trees, graphs, cubes) - type systems and schemas: atomic types, structured types (arrays, maps), set-based type systems (?, *, +) - an overview of functional, declarative programming languages across data shapes (SQL, XQuery, JSONiq, Cypher, MDX) - the most important query paradigms (selection, projection, joining, grouping, ordering, windowing) - paradigms for parallel processing, two-stage (MapReduce) and DAG-based (Spark) - resource management (YARN) - what a data center is made of and why it matters (racks, nodes, ...) - underlying architectures (internal machinery of HDFS, HBase, Spark, neo4j) - optimization techniques (functional and declarative paradigms, query plans, rewrites, indexing) - applications. Large scale analytics and machine learning are outside of the scope of this course. |
| Literature | Papers from scientific conferences and journals. References will be given as part of the course material during the semester. |
| Prerequisites / Notice | This course, in the autumn semester, is only intended for: - Computer Science students - Data Science students - CBB students with a Computer Science background Mobility students in CS are also welcome and encouraged to attend. If you experience any issue while registering, please contact the study administration and you will be gladly added. Another version of this course will be offered in Spring for students of other departments. However, if you would like to already start learning about databases now, a course worth taking as a preparation/good prequel to the Spring edition of Big Data is the "Information Systems for Engineers" course, offered this Fall for other departments as well, and introducing relational databases and SQL. |
Performance assessment
| Performance assessment information (valid until the course unit is held again) | |
Performance assessment as a semester course | |
| ECTS credits | 8 credits |
| Examiners | G. Fourny |
| Type | session examination |
| Language of examination | English |
| Repetition | The performance assessment is only offered in the session after the course unit. Repetition only possible after re-enrolling. |
| Mode of examination | written 150 minutes |
| Additional information on mode of examination | Working on the exercises is rewarded in the sense of ETH's continuous performance assessment with up to 0.25 bonus points. In principle, it is expected that students solve all exercises. In order to control this, every week, 2 to 3 batches of small assignments (25 in total) will be marked as "bonus point eligible". Students can hand in their solution to these special assignments every week, and they will be graded (pass/fail). Each one of the 25 assignment batches gives 0.01 extra point at the exam, which means that up to 0.25 can be obtained as a bonus. Note that these extra points are added before rounding/truncating to a grade with a quarter-point precision and thus may or may not affect the final grade. The final exam may be computer-based. |
| Written aids | Keine. |
| Digital exam | The exam takes place on devices provided by ETH Zurich. |
| This information can be updated until the beginning of the semester; information on the examination timetable is binding. | |
Learning materials
| Main link | Information |
| Recording | All lectures are recorded |
| Only public learning materials are listed. | |
Groups
| No information on groups available. |
Restrictions
| Priority | Registration for the course unit is only possible for the primary target group |
| Primary target group | Data Science MSc (261000)
Computational Biology and Bioinformatics MSc (262100) Computational Biology and Bioinformatics MSc (262200) Computer Science MSc (263000) DAS ETH in Data Science (266000) CAS ETH in Computer Science (269000) Computer Science (Mobility) (274000) |
