I’ve joined a student-led seminar with Gabriel Becker in the Statistics department on dealing with large data, primarily in R. Offered to say a few words about large data problems in my field for a few minutes, so making some notes.
Large dataBases
Genebank: $ > 10^8$ sequences. Phylogenetic inference
Morphological databases: Specialized databases with semantic tools (Dahdul et. al. 2010)
NEON observatories data Keller et. al. 2008
Satellite data (Overpeck et. al. 2011)
Micro-tracker data
Distributed Data
Opportunity: Amount of available data is rapidly increasing under requirements of journals and funders (Whitlock, 2011) (Reichman et. al. 2011) (MOORE et. al. 2010).
Citizen science (Silvertown, 2009)
Challenges: Decentralized, non-standard data formats.
Goal: From meta-analyses to synthetic analyses
Large Computation
Simulation studies – replicates and large sample-spaces
Model estimation – Bayesian estimation
Computing on large datasets – too large to download
Predictive modeling – real-time forecasting
References
Dahdul W, Balhoff J, Engeman J, Grande T, Hilton E, Kothari C, Lapp H, Lundberg J, Midford P, Vision T, Westerfield M, Mabee P and Kelso J (2010). “Evolutionary Characters, Phenotypes And Ontologies: Curating Data From The Systematic Biology Literature.” Plos One, 5. https://dx.doi.org/10.1371/journal.pone.0010708.
Overpeck J, Meehl G, Bony S and Easterling D (2011). “Climate Data Challenges in The 21st Century.” Science, 331. ISSN 0036-8075, https://dx.doi.org/10.1126/science.1197869.
Whitlock M (2011). “Data Archiving in Ecology And Evolution: Best Practices.” Trends in Ecology & Evolution, 26. ISSN 01695347, https://dx.doi.org/10.1016/j.tree.2010.11.006.
Reichman O, Jones M and Schildhauer M (2011). “Challenges And Opportunities of Open Data in Ecology.” Science, 331. ISSN 0036-8075, https://dx.doi.org/10.1126/science.1197962.
MOORE A, Mcpeek M, RAUSHER M, RIESEBERG L and WHITLOCK M (2010). “The Need For Archiving Data in Evolutionary Biology.” Journal of Evolutionary Biology, 23. ISSN 1010061X, https://dx.doi.org/10.1111/j.1420-9101.2010.01937.x.
Silvertown J (2009). “A New Dawn For Citizen Science.” Trends in Ecology & Evolution, 24. ISSN 01695347, https://dx.doi.org/10.1016/j.tree.2009.03.017.
A Continental Strategy For The National Ecological Observatory Network, Michael Keller, David S Schimel, William W Hargrove, Forrest M Hoffman, (2008) Frontiers in Ecology And The Environment, 6 10.1890/1540-9295(2008)6[282:ACSFTN]2.0.CO;2