Strange behavior from the cut function with dates in R

Update: @hadlywickham tweeted to me to let me know that  daylight savings time was the culprit. Though this explains the behavior I document in the first part of this post, the behavior of the cut function using truncated dates (discussed further down the post) is still unexplained.

I recently encountered some strange behavior from R when using the cut.POSIXt method with “day” as the interval specification. This function isn’t working as I intended and I doubt that it is working properly. I’ll show you the behavior I’m seeing (and what I was expecting) then I’ll show you my current base R workaround. To generate a reproducible example, I’ll use this latemail function I gleaned from this stack overflow post.

latemail <- function(N, st="2013/01/01", et="2013/12/31") {
 st <- as.POSIXct(as.Date(st))
 et <- as.POSIXct(as.Date(et))
 dt <- as.numeric(difftime(et,st,unit="sec"))
 ev <- sort(runif(N, 0, dt))
 rt <- st + ev
 }

And generate some data…


set.seed(7110)
#generate 1000 random POSIXlt dates and times
bar<-data.frame("date"=latemail(1000, st="2013/03/02", et="2013/03/30"))
# assign factors based on the day portion of the POSIXlt object
bar$dateCut <- cut(bar$date, "day", labels = FALSE)

I expected that all rows with the date 2013-03-01 would receive factor 1, all rows with the date 2013-03-02 would receive factor 2, and so on. At first glance this seems to be what is happening.

head(bar, 10)
     date                 dateCut
1    2013-03-01 19:10:31  1
2    2013-03-01 19:31:31  1
3    2013-03-01 19:55:02  1
4    2013-03-01 20:09:36  1
5    2013-03-01 20:13:32  1
6    2013-03-01 22:15:42  1
7    2013-03-01 22:16:06  1
8    2013-03-01 23:41:50  1
9    2013-03-02 00:30:53  2
10   2013-03-02 01:08:52  2

Note that at row 9 the date changes from March 1 to March 2 and the factor (dateCut) changes from 1 to 2. So far so good. But we shall see some strange things in the midnight hour.  
Continue Reading →

6

Insert random NAs in a vector in R

I was recently writing a function which was going to need to deal with NAs in some kind of semi-intelligent way. I wanted to test it with some fake data, meaning that I was going to need a vector with some random NAs sprinkled in. After a few disappointing google searches and a stack overflow post or two that left something to be desired, I sat down, thought for a few minutes, and came up with this.

#create a vector of random values
 foo <- rnorm(n=100, mean=20, sd=5)
#randomly choose 15 indices to replace
#this is the step in which I thought I was clever
#because I use which() and %in% in the same line
 ind <- which(foo %in% sample(foo, 15))
#now replace those indices in foo with NA
 foo[ind]<-NA
#here is our vector with 15 random NAs 
 foo

Not especially game changing but more elegant than any of the solutions I found on the interwebs, so there it is FTW.

9

GIS in R: Part 1

I messed around with R for years without really learning how to use it properly. I think it’s because I could always throw my hands up when the going got tough and run back and cling the skirts of Excel or JMP or Systat. I finally learned how to use R when I needed to do a fairly hefty GIS project and I didn’t have access to a computer with ArcGIS and couldn’t afford to buy it (who can?). So I started looking into R’s spatial abilities.

Admittedly R might not be the most obvious choice for free GIS options, combinations of QGIS (http://www.qgis.org/), GRASS (http://grass.osgeo.org/), PostGIS (http://postgis.refractions.net/), or OpenGeo (http://boundlessgeo.com/solutions/opengeo-suite/download/) might pop up in google searches before R. R might not even be the first general purpose programming language you think of for GIS, especially now that ArcGIS relies on Python for much of its modeling. However, all of these tools have a significant learning curve, and I was farther along in R than any of these alternatives, so I started googling and watching tutorial videos. So should you be using R for analyzing and displaying spatial data? If you already know a little or a lot of R, if you need a cross platform solution, or need to do some fairly heavy stats applications to your spatial data, R just might be a good solution for you. It turns out R has lots of support for spatial data and does a great job displaying it too.

There are a number of packages useful for analyzing and displaying your spatial data. I think the 4 most useful right out of the gate are sp, rgdal, maptools, and raster. If you haven’t installed packages before do this…

install.packages(“sp”)
install.packages(“raster”)
install.packages(“maptools”)

…and if you are on a Windows machine…

install.packages(“rgdal”)

If you’re on a Mac, installing rgdal is a little tricky. Give this a try

setRepositories(ind=1:2)
install.packages(“rgdal”)

If that doesn’t work read this over.

http://blog.fellstat.com/?p=46

After installing the packages, if you want to use the functions contained in that package you need to load the library. To use the functions in the sp package, you should type

library(sp)

to load the rgdal package…

library(rgdal)

etc.

0

Science… sort of DC Meetup 2014!

Science_Club_logoWe’ve noticed through the magic of download statistics that we have a nice cluster of listeners in the DC metro area. With Patrick always nearby and Ryan in town for a bit, we figured a meet up would be fun. So on Thursday, January 9th, starting around 7:30pm EST we’ll be convening at the Science Club bar to  hang out, meet some new people, and share some drinks. We hope that you’ll join us if you can.

More details about the event can be found on our Eventbrite page here: Science… sort of DC Meetup 2014

For those who RSVP on the Evenbrite page, your first drink of the evening is on us. See y’all soon!

 

0

Winter Solstice Survival Guide 2013

The most wonderful time of the year, obviously.

The most wonderful time of the year, obviously.

’tis the season to consume! But for this part of the year, as the Winter Solstice approaches, we consume not for ourselves, but for others, and that’s generally a good thing. Each year we here at Science… sort of like to put together a list of suggestions for the science-inclined in your life. Whether you’re looking to give the gift of science, or fleshing out your own list to send to your regionally-appropriate gift giving elf-spirit, this list should have you covered.

We begin with Patrick, who really went above and beyond with contribution, earning him well-deserved top billing.

Continue Reading →

0

Science link fest for the week of the 27th

Hello Paleoposse! This week I bring you Egyptians and iron meteorites, Neil deGrasse Tyson as the Carl Sagan of our generation and a mild rant about poor quality science reporting. I’m off to enjoy my 30th birthday. Hope you all have an awesome weekend!

Solid science:

  • Here’s a story that’s near and dear to my heart: the first use of iron in Egyptian history came from a meteorite. I really love this article as it’s the perfect combination of science and history and it’s written (mostly) free of jargon. And it’s free through Meteoritics and Planetary Science
  • Continuing with the news on space stuff, this article from Scientific American talks about Comet ISON and how studying it can yield clues about the formation of our solar system.
  • One of my favorite geologists, and personal friend, Dana Hunter, has a great story about the glacier-sculpted geology of Discovery Park in Seattle. She focuses mostly on the South Bluff, which is this thick with layers of mud and sand that record the formation of the Sound. And there’s a picture of the back of my head from my last trip when she gave a tour of the area.

Continue Reading →

0

Your science links for the week of July 19th

Howdy paleoposse! Here are this weeks sciency links for your weekend reading pleasure.  This weeks links cover gold producing neutron stars, the boredom of travelling to Mars, some words of inspiration from Richard Feynman, and Comic-Con 2013.

Things that are science:

0

Science link goodness for the week of July 12th

Hello paleoposse! This week I bring you some of my favorite science articles from around the internet. In keeping with the podcast theme, I’ve divided it up into things that are science, things that are sort of science, and things that wish they were science. This week we cover everything from chondrules to volcanoes to the science of the Flash and even a crazy preacher.

Things that are science…

  • One of the more enduring controversies in meteoritics is how the rounded silicate inclusions in chondrites, chondrules, were produced. Some have argued that they condensed from the early solar nebula, while others contend that they originated from the guts of the earliest exploded planetesimals. The latter explanation is gaining momentum.
  • Slate has an excellent article on why testicles are kept in such a vulnerable position on most mammals.
  • National Geographic brings us some gorgeous photos of the Mexican volcano, Popocatepetl.
  • This article from BBC News looks at the science of pessimism and how one can become an optimist. Take some of it with a grain of salt, but still a fun read.
  • Astronomers are working to figure out the origin of radio bursts that occur nearly every second every day. Some proposed ideas are evaporating black holes (my personal favorite) or black holes devouring neutron stars.
  • Forget the Six Million Dollar Man. Bring on the new super food, the five million dollar broccoli! 

Things that are sort of science…

And things that wish they were science…

  • In keeping with the “things that wish they were science” theme, io9 has a post about a preacher that’s convinced the new Star Trek movie will lead humans to really love their animals.
0

Why are Birds Dinosaurs?

nationalgeographic.com

Month after month, one of the most popular posts on the Paleocave blog is the How to Read a Cladogram post I did some time ago. I always intended to follow it up with more cladistic fun. So, hold onto your butts, we’re going to let the dinosaurs loose.

Birds are dinosaurs. We’ve all heard this. But does that phrase make any sense? Not really. Dinosaurs, for the most part, are things that were really big, were mostly scaly, had fantastic teeth, and are extinct. Birds, on the other hand, don’t have teeth, are generally small, and are covered in feathers (I know that you know that lots of old school dinosaurs had feathers too, but whatever). So, why do we say that birds are dinosaurs? The answer involves evolution and the meaning of taxonomic names in biology.

Continue Reading →

4

Powered by WordPress. Designed by Woo Themes