I am trying out Jekyll as an alternative to WordPress for powering this site and my open lab notebook. If you subscribe, the new RSS (atom) feed for the new notebook is at http://carlboettiger.info/atom.xml. I’d love to hear any feedback, good or bad, about the switch.  A recent post gives my own reflections.  The site’s source-code is available on github.

Navigating two platforms

The WordPress site now lives at http://carlboettiger.info/wordpress, while the new Jekyll site lives at the root domain: http://carlboettiger.info. All former links to the wordpress site will still work (though they are also available with “wordpress” in the url).  Enjoy the faster, simpler site.

Warning Signals

Warning signals paper accepted. Making minor edits before submitting final copy.
Added Alan’s edits.

• Review model section again
• Letter: Response to reviewers
• Create Diff
• Media Summary
• Submit

Done!

Database papers

• rfishbase edits made, reformatting in word, sent to Peter
• Ethics form, recommended reviewers, cover letter.
• Submit

Done!

Misc

NERSC web directory: create www in /project/projectdirs/m1366

NSF Arctic SEES grants

These larger, integrated efforts would promote understanding of Arctic systems and would develop optimized models, multiple stable state scenarios, sustainable pathways, decision matrices, visualization techniques and data infrastructure to aid decision making and communication, and structural, energy and communications technology solutions which would inform community practices, management, and policy for a more sustainable Arctic environment.

Methods in Ecology & Evolution is publishing quite a few R packages recently.

• abc
• RNetLogo
• smatr
• BaSTA
• mvbund
• phytools

Catching up on notebook entries for last week and the beginning of this week in one go here. In the process of wrapping up a variety of older projects, so this is kinda all over the place.

rfishbase

• Create function table,
• finalize functions
• Add support for references. Done, see getRefs. (And isn’t it awesome that you can link to a specific linenumber (i.e. line 280) in github by adding #L280 to the end of the url?)
• Picked up some great grep tools in the process thanks to stackoverflow: handling empty matches and multiple matches
• No luck in setting templates for pandoc’s conversion to word. Should be possible with --reference.docx but no luck. (meanwhile, who knew .docx and .odt were just zip archives you could unzip and poke around the xml? Unfortunately styles.xml is a bit opaque to me).

treebase

• Replace all sapply calls with dedicated functions
• Table of functions
• Simplify presentation
• markdownify the Sweave file.

Prosecutor’s fallacy

• Running examples

Wrightscape

• Collecting, organizing and selecting examples into knitr format from demo files.

StatCompSci

• Duncan is running a brown bag seminar on big data computing. We’re coordinating all material over on github.

Other

• Write ievobio proposals
• prsb review — Done.
• Experimenting with a Jekyll based site using CSS from twitter-bootstrap.
• Coordinating Academic Publishing panel discussion

Misc code tricks

• clone all branches
• image alignment in markdown
• markdown in diqus

ruby versions can drive you nuts sometimes.

• Getting out of 1.8.7 and into 1.9.2
• Redcarpet isn’t the default interpreter, but works better. However, it’s latest version has issues.

• rfishbase updated to Duncan’s feedback
• writing intro section on Prosecutor’s Fallacy.
• wrightscape update project repository. Scripts mostly in demos, results in notebook. Notebook makes is easy to get the most recent writeup on the topic, from where I can start to track back the history, but doesn’t provide a good at-a-glance view of the analyses I’ve run side-by-side. Moving the R script demos/ into markdown knitted files in inst/examples/ and rerunning. See directory.
• Files that require cluster computation won’t be able to upload graphs directly. Should separate these scripts from their respective plotting scripts.

Working examples

• Have labrid_data, labrid_mc, focal_examples, simulated_release working. See directory. Running the direct model comparisons on the other traits.

Data management

• Trying out the API for thedatahub.org for storing data. Starting with the python tool, some difficulty getting this running. Detailed issue log
• Also put in form to get www hosting through NERSC.
• Meanwhile, am just tossing data into the webserver running on our local lab computers. This just provides a home for the data so I don’t have binary files on github — the metadata for these objects is handled by the README that lives with the code.
• For large data files (which I’m not generating in these codes) the clear way forward is storage you can send the computation to, rather than downloading. For above solutions work for users of our cluster, or of NERSC, but for a more general audience AmazonS3 is the obvious choice (though involves monthly fees). Looks like there is already a good way to put this into my workflow using RAmazonS3

First install libraries for Ubuntu:

sudo apt-get install libmime-base64-urlsafe-perl libdigest-hmac-perl libdigest-sha1-perl

Then install

install.packages("RAmazonS3", repo="http://www.omegahat.org/R")

And then we’re ready to roll. See Duncan’s tutorial

Will consider a tutorial for this + maybe Segue for Duncan’s seminar. (Note that Duncan’s moved lots of omegahat.org onto github, which we’ll be using for all the class content.)

Reading

• Adaptive Management Conference Series
• PLoS direct submission to wiki
• Tom Preston-Werner talk
• Open CPU
• Humans aren’t good random number generators. uniform frequency example and game in which a machine-learning algorithm exploits your non-randomness

Misc

modify knitr hook for formatting inline code, e.g.

knit_hooks$set(inline = function(x) {
 prettyNum(round(x,2), big.mark=",")
})

• Avoid knitr displaying X11 during interactive runs
• Refresh cache

Sweave is something of a gold standard in reproducible research. It creates a dynamic document, written in a mix of LaTeX and R code where the results of the analysis (numbers, figures, tables) are automatically generated from the code and inserted into the resulting pdf document, making them easy to update if the data or methods change. It’s a nice idea, in principle.

However, the practical troubles are many. Coauthors don’t know LaTeX, publishers who don’t accept LaTeX or pdfs. The LaTeX myth that you are freed from thinking about formatting, when in fact you have to fill your document with LaTeX specific markup that makes it a burden both to type and to read the source-code. Compiling and debugging your text. And then the reproducibility comes from sharing that Sweave file — a mix of LaTeX and R that almost no one can read easily. Where’s the elegance in that? ¹ Sure, none of these are show-stoppers — I’ve been content with LaTeX for years — but suddenly there’s a better way.

Thanks to knitr, a successor of Sweave, I can write my publications in markdown. Unlike LaTeX, HTML, or other markup languages, markdown is designed to be easily read as plain text, but can also be interpreted into pretty HTML, and now into almost any other format thanks to pandoc. All of which is to say that writing and sharing just got a lot easier.

As I have written previously, I already use this markdown format for my notes and code, so there’s no re-typing required. When working on the paper, I can just write. I can edit the code without flipping back and forth between files. Knitr can run the code blocks, caching parts that have already run for efficiency, and upload the resulting figures in png format automatically to the Internet. Github displays the resulting document and the /source, while also tracking the versions as my writing progresses.

Pandoc allows me to transform these notes into a LaTeX file that can generate professional-looking pdfs with given journal .cls files by using a custom latex template. Pandoc can also generate the less pretty but often required word documents. A separate Rscript combines with a Makefile to control the relevant formatting — for LaTeX output I want high-quality pdf graphics, for Word-doc output I want eps graphics which are created but not pasted into the Word file, for the drafts I want png graphics stored online for easy sharing. Pandoc allows citations to be extracted from my Mendeley library (via Bibtex files) and inserted into each of these output formats (doc, pdf, github markdown).

Getting the LaTeX template, Makefile, and knit script set up for this pipeline takes a little care — mostly to ensure figures and tables look appropriate in all outputs. Once these files are created though, they can be easily reused on other manuscripts. A simple make pdf builds the pdf copy, make docx builds a MS Word copy,² and make github the copy that displays with images on Github.

The links in this post point to what is an active draft of a little manuscript at the time of this writing. In addition to making the final result reproducible, Github captures the provenance or history of the research and writing process. It’s not a perfect system, but it’s a nice step.

1. I’m glossing over the additional challenges of highlighting, caching, and formating on the R code side, which have been largely addressed by additional packages and are elegantly solved in knitr.
2. though these binary files aren’t stored in the git repository

Fishbase

• Revisions based on Duncan’s feedback
• Formatting to Journal of Fish Biology guidelines
• Updated formatting

1. fixed footnotes issue in pdf version.
2. Externalize tables in word version.

    knit_hooks$set(chunk = function(x, options) {
      if (!options$split) return(x)
     cat(x, file = fig_path('.md', options))
    }

* Consider more intuitive function names rather than function arguments?

TreeBase

• Needs additional functions replacing sapply queries and argument queries.
• Table of functions
• re-organize presentation

rOpenSci GSOC

GSoC applications were due at noon today. Provided final rounds of feedback to our three applicants. Proposals (secure):

• Christoph
• Vijay
• Artem

Reading

Berkeley adaptive management / resource economics

• Michael Hanemann
• Stephanie Carlson? (Evolution/harvesting, Mangel graduate)
• Larry Karp?

Also looking at Mangel’s Decision-making and investment under uncertainty, particularly within a Bayesian framework, and ecological detection, with application to adaptive management.

• “Ecology, Conservation and Public Policy: A Vision for the 21st Century”, Astor Lecture, University of Oxford, April 2007 slides
• Ecosystem management perspective (http://dx.doi.org/10.1126/science.1098222 ), (http://dx.doi.org/10.1525/bio.2009.59.8.11), (http://dx.doi.org/10.1098/rstb.2004.1571)
• Adaptive management NCEAS synthesis (http://dx.doi.org/10.1002/(SICI)1520–6602(1998)1:1<16::AID-INBI3>3.0.CO;2-D)
• nice treatment of stochastic transitions in (http://dx.doi.org/10.1006/tpbi.1994.1002)

Other reading

• Tara Martin, CSIRO’s Conservation Decisions Lab
• Paul Fackler’s MDPSolve

Bibliography

Really excellent talk by Professor Michael Jordan from the Berkeley Statistics department visiting us at Davis yesterday. Haven’t seen a talk cover technical content with remarkable clarity for the algorithms and insights involved, while also covering such a breadth of material. My notes didn’t really keep up, but transcribing some of my scribbling into electronic form will have to wait until I have more time. (Their pubs are probably a better reference anyway.)

But to summarize,I enjoyed the bag of little bootstraps — a very simple, elegant, and powerful idea. Divide-and-conquer meets Efron.  (http://arxiv.org/abs/1112.5016)

I hadn’t realized NetFlix prize could be thought of a matrix completion problem with justifiable assumption of low-rank, and was impressed both by how simple the best algorithm is and by the 10 fold speedup obtained by their approximate divide-and-conquer solution.  (http://arxiv.org/abs/1201.6002)

Highlights the use of Stein’s method of exchangeable pairs for giving you a differential-equation approach to proving things like central limit theorem, and a remarkably simple proof for a Bernstein-like large deviation result for matrices sounds pretty clever.

Bibliography

knitr + markdown + pandoc + latex

Troubleshooting this workflow still…

• Fonts
• xtable for html & latex/pdf
• xtable comments display
• tikz device trouble
• Pandoc table trouble
• pandoc footnotes
• Getting a satisfactory LaTeX template.
• Makefile

writing writing writing:

Mostly writing this week, so my notebook has been lacking in entries. Trying to move several manuscripts forward: two are papers accompanying software packages for extracting, manipulating and visualizing data from online repositories (TreeBase and FishBase); part of the rOpenSci project. These databases are great, but it difficult to pursue really interesting questions until we can spin it around on our fingertips. Web-based repositories make the data easily accessible but not easily manipulable. The R language provides a powerful and widely used platform to explore, visualize, and manipulate data once we have it. These packages aim to bridge the gap between access and use, between the web, and R.

• treebase manuscript
• rfishbase manuscript

Other tasks for the week

• prosecutor fallacy writing.

An exciting day for Big Data (particularly if your Berkeley).

NSF

• Encouraging research universities to develop interdisciplinary graduate programs to prepare the next generation of data scientists and engineers;
• Funding a $10 million Expeditions in Computing project based at the University of California, Berkeley, that will integrate three powerful approaches for turning data into information — machine learning, cloud computing, and crowd sourcing;
• Providing the first round of grants to support “EarthCube” – a system that will allow geoscientists to access, analyze and share information about our planet;
• Convening researchers across disciplines to determine how Big Data can transform teaching and learning.

DOE

The Department of Energy will provide $25 million in funding to establish the Scalable Data Management, Analysis and Visualization (SDAV) Institute. Led by the Energy Department’s Lawrence Berkeley National Laboratory,

DoD

• The Department is seeking a 100-fold increase in the ability of analysts to extract information from texts in any language.
• The Defense Advanced Research Projects Agency (DARPA) is beginning the XDATA program, which intends to invest approximately $25 million annually for four years to develop computational techniques and software tools for analyzing large volumes of data. The XDATA program will support open source software toolkits to enable flexible software development for users to process large volumes of data in timelines commensurate with mission workflows of targeted defense applications.

USGS

USGS is announcing the latest awardees for grants it issues through its John Wesley Powell Center for Analysis and Synthesis. The Center catalyzes innovative thinking in Earth system science by providing scientists a place and time for in-depth analysis, state-of-the-art computing capabilities, and collaborative tools invaluable for making sense of huge data sets. These Big Data projects will improve our understanding of issues such as species response to climate change, earthquake recurrence rates, and the next generation of ecological indicators.

NSF’s email alerts today reflect this initiative with more details:

• Cyberinfastructure track comes to IGERT training
• Data intensive education related research
• Big Data
• Data Citations

I’ve just been accepted to the Data to Knowledge Conference in Berkeley this May.  I think this conference will address some of the key issues facing us in Big Data today across many different fields, and I hope to learn much that could be useful to ecology and evolution.  Given the audience I look forward to being a source of questions rather than answers as I put on my biologist’s cap.  Berkeley’s particular leadership in this area is probably evident in the millions of dollars they are receiving from the various initiatives coming out of today’s Big Data announcement.  What better day to apply?

My Abstract

Many complex systems, from ecosystems to economies to governments, can undergo sudden and often catastrophic transitions between alternate stable states. Ecologists have recently begun to demonstrate the existence of certain early warning signals or leading indicators — patterns that suggest an ecosystem is approaching a critical transition such as the collapse of a population or eutrophication of a lake. Though these patterns are extremely subtle and consequently require large amounts of data to generate an adequate ratio of signal to noise, technology such as satellite imaging and microchip sensors have dramatically increased the data available. Will this recent data deluge help us detect critical transitions with enough time avert them?

I will briefly introduce recent advances in the field of early warning signals and discuss my work to transition this from a proof of principle to a quantifiable risk that can inform policy.