Skip to main content

ICS 411 Big Data Storage and Processing

The field of computer science is experiencing a transition from processing-intensive to data-intensive problems, wherein data is produced in massive amounts by large sensor networks, simulations, and social networks. Efficiently extracting, interpreting, and learning from these very large data sets need different storage and processing requirements compared to traditional business applications that are mostly dependent on relational database management systems. These emerging data-intensive applications require heavy read/write workloads and do not need some of the stringent schema and ACID properties that are central to relational databases. To cope with these requirements, a new genre of large-scale systems, is introduced that is called NoSQL databases. The main characteristics of NoSQL databases are that they are open source, non-schema oriented, having weak consistency properties and heavily distributed over large and clusters of commodity hardware. In this course, we will cover the basic concepts and approaches that are used by such big-data systems. Students will gain hands-on experience by solving relevant problems through projects utilizing publicly available systems. Topics covered includes: fundamentals of big data storage and processing using Hadoop, distributed file systems, and map-reduce, fundamentals of the four categories of NoSQL systems, namely kay-value stores, document stores, column stores, and graph stores. Students will implement applications using the following systems: Apache HBase, Amazon's Dynamo, Apache Cassandra, MongoDB, and Neo4J.


Special information

Note: Students are responsible to both be aware of and abide by prerequisites for ICS courses for which they enroll, and will be administratively dropped from a course if they have not met prerequisites.First day attendance is mandatory.
4 Undergraduate credits

Effective May 3, 2017 to present

Learning outcomes


  • Identify and justify the storage and processing requirements of data-intensive applications.
  • Explain the similarities and differences between the requirements of big-data applications and the ACID requirements of traditional database applications.
  • Analyze and solve data-intensive problems using Hadoop and the distributed file system.
  • Design and develop algorithms using the map-reduce programming paradigm.
  • Classify and describe NoSQL systems
  • Assess the suitability for using a particular type of NoSQL databases for an application
  • Experiment, contrast and evaluate the following NoSQL systems: MongoDB, DynamoDB, Cassandra, HBase, and Neo4J.