Thesis Outline and Additional Papers

See research for previous writing

0. Front Matter
   1. Title
   2. Abstract
   3. ToC, LoF, LoT
1. Introduction
   1. Broad introduction to topic
      • Proposal
   2. Problem Statement
      • Proposal
      • Project Overview
      • Means of development
      • Emphasise developer level
      • Match to what has been produced (work backwards a little)
   3. Beneficiaries of this research
   4. History & context
      • big data work group
2. Literature Review
   1. Overview: review method, limitations, scope
      • Literature Review
   2. Literature, grouped by similarities
      1. Packages
         • A Review of Iteration with Sparklyr
         • A Review of pbdR
         • A Review of foreach
         • A Disk.Frame Case Study
         • A Detail of Future
      2. Algorithms
         • A Detail of Reweighted Least Squares
         • Distributed Hash Table for the Dependency Graph
3. Brief theory chapter
   1. Motivating the following chapter’s aspects, objectives which will be linked in the next chapter to each aspect
   2. Defining a distributed statistical modelling system for R
      • What does it mean to be distributed?
      • Properties that distributed object/chunk/computation etc. must possess
      • What can’t be done
      • Direction: Statistical Modelling
      • High-level (like mathematical definition)
      • Response
4. Illustrative Problem
   1. Aspect: Object System
   2. Aspect: Computation
   3. Aspect: Concurrency
   4. Aspect: Reference
   • further misc. notes on Communication
   • further misc. notes on object system
   • further misc. notes on concurrency
   • Definitive System Aspects
5. Experiments in implementation of System
   • Motivating the following outline of eventual implementation (Tying in with aspects from previous chapter)
   • Bring over thoughts on implementations of concurrency, carried over from the concurrency aspect section in the previous chapter
   1. RServe-based System - Current and Proposed Information Structures - Experiment: Eager Distributed Object Supplement - Experiment: Eager Distributed Object Precursory Report - Experiment: Distributed Decision Tree
   2. Redis-based System - Initial Distributed Object Experimentation with a Message Queue Communication System - Report on Current Chunk Architecture - Chunk ID Origination and Client-Server Communication - Message Queues for Communication in a Distributed Object System - Initial Chunk Experimentation with a Message Queue Object System - Inter-node communication with Redis
   3. DistObj System - distObj System Initialisation and Input - DistObje Non-Assigned Data Return - Description of distObj Client-Server Call Process
6. Outline of System Implementation
   • API and Architecture
   0. Overview
      • A Description of the New System Architecture
      • Project Overview
      • Layering
   1. orcv
      • API, architecture
      • Message Queues for Communication in a Distributed Object System
      • Necessity of multithreading?
   2. chunknet
      • API, architecture
      • largerscale REST API
      • Models of Communication in the LargerScale System
      • Introduction to largerscale Objects
      • Acyclic Dependency Dependency Digraphs for Immutable Chunks
      • Discussion on the Chunk Building Block
      • Asynchronout Server-Side Resolution Monitoring
      • Chunk ID Origination and Client-Server Communication
      • Aspect: Mutability & Garbage Collection
   3. largescaler
      • API, architecture
      • Value of Functional & Higher-Order functions
      • Discussion on the Chunk Building Block
      • Distributed Reduce in LargeScaleR
   4. largescalemodelr
      • API, architecture
      • dlm: Distributed Linear Models
7. System Capabilities
   1. Application
      • ADMM
      • lm, gml, admm, xgboost, boosting
      • What is more efficient, what is less (data movement)
   2. Theoretical comparison with other Systems
      • A Comparison of Work Sequences
   3. Benchmarks
      • e.g. Against Spark table, foreach etc.
      • Validity and justification of benchmarks
   4. Extensions to the system
      • e.g. shuffle, index as in-system extensions
      • Open/Closed principle
      • Extensions serving to validate the system
8. Recommendations
   1. Comparison between expectation and reality
      • Limitations - Evaluation & Scope
   2. Future Work
      • Self-healing Data in Largerscale
      • Distributed Hash Table for the Dependency Graph
      • Acyclic Dependency Dependency Digraphs for Immutable Chunks
      • Application Performance Management
9. Appendices
   1. Documentation
   2. Source Code
   3. Bibliography