An opinionated view of the Tidyverse “dialect” of the R language, and
its promotion by RStudio.
Norm Matloff, Prof. of Computer Science, UC Davis (former Prof. of Statistics at UCD)
Note: This essay is somewhat frank, involving the very
popular Tidyverse and RStudio. I hope it is polite and taken as
constructive criticism.
I like and admire the RStudio people, including the Tidyverse
originator, Hadley Wickham, and have always supported them, both
privately and
publicly. I have been interacting with them from the
beginning, when the firm consisted of only founder JJ Allaire and ace
developer Joe Cheng. I highly praise the firm to my students, and I use
and recommend Hadley’s ggplot2 package (though I don’t consider it
part of the Tidyverse, having been developed well before Tidy and
thematically unrelated).
In other words, I absolutely don’t consider RStudio to be some evil
cabal. I state at various places in this essay that I think their
actions have been well-intentioned. Nevertheless, I believe that
RStudio took a wrong turn when it decided to promote the Tidyverse,
which has led to a situation in which the very health of the language is
at stake.
My bio is here.
Specifically in terms of R, I’ve been an R user and developer since near the
beginning, having used R’s predecessor S before that. I’ve
published several books that use R, and am currently (2019) the
Editor-in-Chief of the R Journal. (Hadley is a former EiC on the
journal.)
-
The Tidyverse arose from the desire to have a set of packages that
are consistent with each other, a “purist” philosophy that appeals,
for instance, to computer scientists. The Tidyverse also borrows from
other “purist” computer science philosophies, notably functional
programming. -
Unfortunately, the price of purity is (a) increased complexity and
abstraction, making code more prone to error, as well as (b) a
sacrifice in performance. Ironically, though consistency of interface
is a goal, new versions of Tidy libraries are often incompatible with
older ones, a very serious problem in the software engineering world. -
In heavily promoting the Tidyverse, especially in the education
realm, RStudio, with its dominance in the R field, is developing an
entire new generation of R users whose skills in base-R are superficial
at best, and who — most importantly — feel that R is the Tidyverse. -
Regardless of being well-intentioned, RStudio is molding R into its
own desired image. That new generation will come to dominate the
community, treating Tidyverse as the “real” base, and viewing the actual
base-R as something akin to assembly language. This might be fine if
the R community were unified in viewing Tidyverse as a high-level
improvement, but many do not; they are not fans of the syntax and so on,
and as noted, are worried about the slow performance of Tidy. -
That new generation will often be biased against non-Tidyverse job
seekers, non-Tidyverse CRAN packages, and academics who submit
non-Tidyverse data science research papers and grant proposals. They
will have no choice but to bend to RStudio’s wishes. Then
RStudio will have succeeded in an end-run around the governing
body of the R language. -
For the above reasons, RStudio is essentially operating as a
monopolist (“essentially” because it is an unorthodox market). Though
again this is well-intentioned, the adverse impacts are grave. -
One key example of the pernicious effects of this monopolistic
situation is that RStudio’s promotion of the Tidyverse has alarmingly
impeded the progress and adoption of technologically superior packages,
notably data.table. Most in the “Tidyverse generation” are unaware
of anything outside the Tidyverse. -
RStudio’s development of the Tidyverse is a good thing, and those who
like its philosophy should use it. My objection is in Point 5 above.
I would give as an example the fact that R has various object-oriented
programming (OOP) paradigms to choose from, such as S3, S4 and R6. I
think it’s great that, e.g., R6 is available, but I don’t want to be
forced to use it. (Just an example; both the Tidyverse and I use S3.) -
A major reason offered by RStudio for promoting the Tidyverse is that
it makes R easier to teach to non-programmers. I would argue that the
Tidyverse makes R harder to learn for this group. -
RStudio can easily remedy the situation. I have recommendations at
the end of this essay.
The dplyr package is a featured app of the Tidyverse developed by
Hadley, so I’ll use this as an example at several points in this essay.
Dplyr borrowed a number of ideas from the earlier data.table by
Matt Dowle. One of Hadley’s major motivations was to give the user a
more “English-like” interface. (Note: I regard both dplyr and
data.table as advanced topics; neither is suitable for beginners.)
Unfortunately, dplyr is much, much slower than
data.table on large datasets. Here are some of the
timing examples, for various operations, on the
H2O site
(times in seconds; see above URL for details):
| dplr | data.table |
| 37.3 | 9.07 |
| 95.5 | 9.20 |
| 496 | 11.9 |
The differences are even starker in
this study
by the consulting firm Win-Vec LLC, e.g.
showing that dplyr can be extremely slow even relative to base-R,
thus even worse relative to data.table.
RStudio, a commercial entity, is heavily engaged in educational
activities, both with its own courses and in interacting with teachers
of R. It has heavily promoted dplyr. Meanwhile, RStudio and its
allies have typically ignored data.table in these promotions, e.g.
not mentioning it in their timing comparisons, and painting it as beyond
the reach of nonprogrammers.
Hadley did belatedly develop an interface, dtplyr, in 2016,
2 years after introducing dplyr and 8 years
after data.table. And even this is
slow.
RStudio is after all a for-profit business, so such treatment of
data.table is just good business practice. But R is an open-source
language, and these actions have harmed R. Advocates of other
languages, notably Python, constantly denigrate R as being slow on large
datasets. Actually, data.table is extremely fast (and is faster
than Python’s Pandas), but those who view R through the RStudio lens are
unaware of it.
Teaching has been a keen interest of mine since my college days. I’ve
been a teacher of stat and computers for many years, and have won
various teaching awards. My textbook, Statistical Regression and
Classification: from Linear Models to Machine Learning, was the
recipient of the 2017 Ziegel Award.
But it goes far beyond that; I really am intensely interested in how
people learn, from children to middle-aged adults. Among other things,
I’ve taught English As a Second Language to immigrant adults, most of
whom have had less than a high school education.
The Tidyverse advocates’ claim
From this background, I strongly question the claim made by Tidyverse
advocates that it facilitates the teaching of R to beginning
programmers, as opposed to teaching base-R. (Again, I regard both
dplyr and data.table as advanced topics; neither is suitable for
beginners.)
There has been no study of this claim. Advocates often provide
testimonials from students like “I learned R using Tidyverse, and now am
productive in R!” — which says nothing at all about the teachability of
base-R in comparison. (It is ironic that advocates who present such
statements are statisticians, who ought to know the need for a control
group.)
Tidyverse makes learning harder, not easier
Contrary to the Tidy advocates’ claim, I believe using the Tidyverse
makes things more difficult for learners without prior programming
background.
Tidyverse students are being asked to learn a much larger volume of
material, which is clearly bad pedagogy. See “The Tidyverse
Curse”, in which the
author says inter alia that he uses “only” 60 Tidyverse functions —
60! The “star” of the Tidyverse, dplyr, consists of 263
functions.
While a user initially need not use more than a small fraction of them,
the high complexity is clear. Every time a user needs some variant of
an operation, she must sift through those hundreds of functions for one
suited to her current need.
By contrast, if she knows base-R (not difficult), she can handle any
situation. The old adage applies: “Give a man a fish, and he can eat
for a day. Teach him how to fish, and he can eat for a lifetime.”
Another featured Tidyverse package, the functional programming
(FP)-oriented library purrr, has 177 functions. Again the point
about complexity applies. Even more importantly, top university Computer
Science Departments have shifted away from teaching their introductory
programming courses using functional programming paradigm to the more
traditional Python, as they deem FP to be more abstract and challenging.
It would then seem that using FP to teach non-programmers learning R is
even more unwise.
Similarly, it is bad pedagogy to force students to learn tibbles, a
more complex technology, instead of data frames, a simpler one.
Syntax comparison and teachability
Again, the claim is that the Tidyverse is more teachable because of its
“English-like” syntax, while they dismiss data.table‘s syntax as
opaque.
Below is a comparison (adapted from
here):
We’ll use R’s built-in mtcars dataset.
mtdt <- as.data.table(mtcars) mtdt[cyl == 6] # data.table syntax mttb <- as_tibble(mtcars) filter(mttb,cyl == 6) # dplyr syntax
Is there really any difference? Can’t beginners, even without
programming background, quickly adapt to either one after seeing a few
examples? Even those who claim high teachability for dplyr do
readily agree that their students could also easily pick up
data.table, or for that matter my preference base-R, given some
examples.
And what of the fact that we have the English word filter above?
Granted, it looks nice, but English can be misleading or mystifying in a
computer context. Even an experienced programmer would not be able to
guess what the dplyr function mutate() does, for instance.
The Tidyverse also makes heavy use of magrittr pipes, e.g. writing
the function composition h(g(f(x))) as
Again, the pitch made is that this is “English,” in this case reading
left-to-right. But again, one might question just how valuable that is,
and in any event, I personally tend to write such code left-to-right
anyway, without using pipes:
And much more importantly, even advocates of pipes concede that pipes make
debugging more difficult; by contrast, my style above lends itself easily to
debugging. And again, for large problems, piped code is slower.
Is dplyr needed or even desirable for teaching?
As I said earlier, in discussions with those who report success in using
the Tidyverse to teach beginning programmers, I ask whether their
students are incapable of learning just base-R. They readily concede
that the answer is no. Indeed, before the Tidyverse, throngs of people
were learning base-R without any prior programming background.
The Tidyverse advocates also claim that the “English” in dplyr makes
the code easier to read. To me, that is missing the point; as any
instructor of software engineering can tell you, the best way to make
code readable is to use REAL English, in good, meaningful code comments.
As also mentioned, the Tidyverse can be difficult to debug, and run very
slowly on large datasets.
In short, in my view there is no advantage to teaching R through the
Tidyverse, and some significant disadvantages.
Summary: the proper status of the Tidyverse in teaching
I think it is a mistake to feature the Tidyverse in teaching R, for these
reasons:
-
Complexity and volume.
-
Difficulty in debugging.
-
Inadequate generalizability.
I am certainly not saying one should only use base R; on the
contrary, CRAN is a major advantage of R, which I use extensively.
But the Tidyverse should be considered advanced R, not for beginners,
just as is the case for most complex CRAN packages.
The Long Arm of RStudio
In the SatRday LA conference,
April 6, 2019, a speaker who was actually explaining the advantages of
data.table in large datasets said that package “was created by Matt
Doyle [sic]. Who’s that? No one knows who he is.” He repeated later,
“No one has ever heard of Matt Doyle.” Actually, many in the audience
had indeed heard of Matt Dowle, but in that speaker’s world — the
RStudio-educated world” — his statement about lack of name recognition
for Matt was sadly accurate. Such is the impact of RStudio on the
field.
Hadley’s talk in the 2016 useR! conference
amounted to a manifesto, calling for R to “modernize” along Tidy lines.
He conceded that this would involve “short-term pain,” yet believed it
to be very much worthwhile. Again, given RStudio’s dominance, this
revolution in R was sure to follow, which it has.
The Tidyverse is a vigorous promotional effort by RStudio that has come
to dominate the R world. As mentioned, the firm has been especially
active in the education realm, including gifts of funding and licensed
software, and support for conferences. It runs the Tidy Tuesday “social
event,” offers Tidy tutorials at all R conferences, and so on.
Well-intentioned and useful, to be sure, but with the conscious effect
of increasing the firm’s influence, and some would argue, power.
A good example of the intent of RStudio to bring all of R to the Tidy
world is the broom package. Titled “Convert Statistical Analysis
Objects into Tidy Tibbles,” its goal is to convert the output of
numerous packages into Tidy form.
“Testimonials” are legion. Non-programmers who take Tidy-based R
courses are delighted that they can now do some data analysis, and
praise the Tidyverse without realizing they have no basis for comparison
to base-R.
RStudio counts ggplot2 as being part of the Tidyverse, but it was
developed much earlier, and does not follow the Tidy philosophy. But as
a result of such inclusion, I see many users who, being justly impressed
with ggplot2, mistakenly think that the package can only be used
from Tidy code and thus is an advantage of being Tidy. This illustrates
the mindset that has developed.
And there is a Bandwagon Effect at work, and even a hint of cult-like
behavior. I’ve seen statements on Twitter from “graduates” of Tidyverse
training who actually apologize because their code did not use the
Tidyverse. One post I saw came from a person who panicked because she
had written a for loop rather than employ Tidyverse’s functional
programming package purrr.
And of course many who teach R want to join Bandwagon as well, feeling
they must teach the “latest.”
Adverse Impact
Given these dynamics, we will eventually, maybe rather soon, reach a
point at which most R users will be Tidy, and have indeed “Never heard
of Matt Dowle.” This will make things very difficult for the non-Tidy R
people: Non-Tidy job seekers who are excellent R coders will find that
they are dismissed out of hand by Tidy interviewers; authors of non-Tidy
CRAN code will find their contribution is considered useless; academics
submitting data science research manuscripts or grant proposals will
find that Tidy reviewers give them low scores. In short, R will have to
bend to RStudio’s wishes.
This is classical monopolistic behavior. Of course, the market here,
an open source project dominated by a commercial entity, is uncharted
waters in the legal sense. But in the economic sense, this is exactly
what is occurring, whether a conscious goal of RStudio or not.
Aside from price, again probably not an issue here, a highly insidious
consequence of monopolies is the stifling of innovation. The case of
data.table discussed above is a fine case in point.
An open-source project involves a people spending a large amount of time
developing the project for free, no pay. Thus, for a commercial entity
to then swoop down and exploit all that free labor for its own profit is
fraught with peril. To then take over the product as its own
is unconscionable. I have no doubt that RStudio was well-intentioned in
this, sincerely believing in the Tidyverse, but many do not share this
view, and RStudio should have worked with the R Core Group (see below),
rather than taking action on its own.
The first major firm to become involved in R was Revolution Analytics
(now part of Microsoft). There was much concern in the R community at
the time over Revo’s potential negative impact on R, but instead, they
turned out to be model corporate citizens.
As noted, I know and admire the people at RStudio, but a commercial
entity should not have such undue, unilateral influence on an
open-source project.
It should be noted that neither Hadley nor anyone else one from RStudio
is in the elite, 20-member R Core Group, which controls the development
of the language. The Tidyverse thus is tantamount to an end-run around
the leaders of this open-source project. As any expert in
organizational behavior will tell you, this does not augur well for the
future health of R, in spite of undoubtedly being well-intentioned.
RStudio is a great company, staffed by many talented people. In my
opinion, their one failing has been to run off on their own, rather than
adhering to the norms of open-source projects.
In my view, RStudio can easily remedy the problem. It can take the
following anti-monopolistic actions:
-
Promote the teaching of base-R to beginners, treating the Tidyverse
as an advanced topic. The book, R for Everyone: Advanced Analytics and Graphics (second ed.t), by Jared Lander does exactly this! -
In the various RStudio Web pages on writing fast R code, give
data.table equal time.
