Special symbols
special-symbols.Rd.SD, .BY, .N, .I, .GRP, and .NGRP are read-only symbols for use in j. .N can be used in i as well. .I can be used in by as well. See the vignettes, Details and Examples here and in data.table.
.EACHI is a symbol passed to by; i.e. by=.EACHI.
Details
The bindings of these variables are locked and attempting to assign to them will generate an error. If you wish to manipulate .SD before returning it, take a copy(.SD) first (see FAQ 4.5). Using := in the j of .SD is reserved for future use as a (tortuously) flexible way to update DT by reference by group (even when groups are not contiguous in an ad hoc by).
These symbols used in j are defined as follows.
.SDis adata.tablecontaining the Subset ofx's Data for each group, excluding any columns used inby(orkeyby)..BYis alistcontaining a length 1 vector for each item inby. This can be useful whenbyis not known in advance. Thebyvariables are also available tojdirectly by name; useful for example for titles of graphs ifjis a plot command, or to branch withif()depending on the value of a group variable..Nis an integer, length 1, containing the number of rows in the group. This may be useful when the column names are not known in advance and for convenience generally. When grouping byi,.Nis the number of rows inxmatched to, for each row ofi, regardless of whethernomatchisNAorNULL. It is renamed toN(no dot) in the result (otherwise a column called".N"could conflict with the.Nvariable, see FAQ 4.6 for more details and example), unless it is explicitly named; e.g.,DT[,list(total=.N),by=a]..Iis an integer vector equal toseq_len(nrow(x)). While grouping, it holds for each item in the group, its row location inx. This is useful to subset inj; e.g.DT[, .I[which.max(somecol)], by=grp]. If used inbyit corresponds to applying a function rowwise..GRPis an integer, length 1, containing a simple group counter. 1 for the 1st group, 2 for the 2nd, etc..NGRPis an integer, length 1, containing the number of groups.
.EACHI is defined as NULL but its value is not used. Its usage is by=.EACHI (or keyby=.EACHI) which invokes grouping-by-each-row-of-i; see data.table's by argument for more details.
Note that .N in i is computed up-front, while that in j applies after filtering in i. That means that even absent grouping, .N in i can be different from .N in j. See Examples.
Examples
DT = data.table(x=rep(c("b","a","c"),each=3), v=c(1,1,1,2,2,1,1,2,2), y=c(1,3,6), a=1:9, b=9:1)
DT
#> x v y a b
#> <char> <num> <num> <int> <int>
#> 1: b 1 1 1 9
#> 2: b 1 3 2 8
#> 3: b 1 6 3 7
#> 4: a 2 1 4 6
#> 5: a 2 3 5 5
#> 6: a 1 6 6 4
#> 7: c 1 1 7 3
#> 8: c 2 3 8 2
#> 9: c 2 6 9 1
X = data.table(x=c("c","b"), v=8:7, foo=c(4,2))
X
#> x v foo
#> <char> <int> <num>
#> 1: c 8 4
#> 2: b 7 2
DT[.N] # last row, only special symbol allowed in 'i'
#> x v y a b
#> <char> <num> <num> <int> <int>
#> 1: c 2 6 9 1
DT[, .N] # total number of rows in DT
#> [1] 9
DT[, .N, by=x] # number of rows in each group
#> x N
#> <char> <int>
#> 1: b 3
#> 2: a 3
#> 3: c 3
DT[, .SD, .SDcols=x:y] # select columns 'x' through 'y'
#> x v y
#> <char> <num> <num>
#> 1: b 1 1
#> 2: b 1 3
#> 3: b 1 6
#> 4: a 2 1
#> 5: a 2 3
#> 6: a 1 6
#> 7: c 1 1
#> 8: c 2 3
#> 9: c 2 6
DT[, .SD[1]] # first row of all columns
#> x v y a b
#> <char> <num> <num> <int> <int>
#> 1: b 1 1 1 9
DT[, .SD[1], by=x] # first row of all columns for each group in 'x'
#> x v y a b
#> <char> <num> <num> <int> <int>
#> 1: b 1 1 1 9
#> 2: a 2 1 4 6
#> 3: c 1 1 7 3
DT[, c(.N, lapply(.SD, sum)), by=x] # get rows *and* sum all columns by group
#> x N v y a b
#> <char> <int> <num> <num> <int> <int>
#> 1: b 3 3 10 6 24
#> 2: a 3 5 10 15 15
#> 3: c 3 5 10 24 6
DT[, .I[1], by=x] # row number in DT corresponding to each group
#> x V1
#> <char> <int>
#> 1: b 1
#> 2: a 4
#> 3: c 7
DT[, .N, by=rleid(v)] # get count of consecutive runs of 'v'
#> rleid N
#> <int> <int>
#> 1: 1 3
#> 2: 2 2
#> 3: 3 2
#> 4: 4 2
DT[, c(.(y=max(y)), lapply(.SD, min)),
by=rleid(v), .SDcols=v:b] # compute 'j' for each consecutive runs of 'v'
#> rleid y v y a b
#> <int> <num> <num> <num> <int> <int>
#> 1: 1 6 1 1 1 7
#> 2: 2 3 2 1 4 5
#> 3: 3 6 1 1 6 3
#> 4: 4 6 2 3 8 1
DT[, grp := .GRP, by=x] # add a group counter
#> x v y a b grp
#> <char> <num> <num> <int> <int> <int>
#> 1: b 1 1 1 9 1
#> 2: b 1 3 2 8 1
#> 3: b 1 6 3 7 1
#> 4: a 2 1 4 6 2
#> 5: a 2 3 5 5 2
#> 6: a 1 6 6 4 2
#> 7: c 1 1 7 3 3
#> 8: c 2 3 8 2 3
#> 9: c 2 6 9 1 3
DT[, grp_pct := .GRP/.NGRP, by=x] # add a group "progress" counter
#> x v y a b grp grp_pct
#> <char> <num> <num> <int> <int> <int> <num>
#> 1: b 1 1 1 9 1 0.3333333
#> 2: b 1 3 2 8 1 0.3333333
#> 3: b 1 6 3 7 1 0.3333333
#> 4: a 2 1 4 6 2 0.6666667
#> 5: a 2 3 5 5 2 0.6666667
#> 6: a 1 6 6 4 2 0.6666667
#> 7: c 1 1 7 3 3 1.0000000
#> 8: c 2 3 8 2 3 1.0000000
#> 9: c 2 6 9 1 3 1.0000000
X[, DT[.BY, y, on="x"], by=x] # join within each group
#> x V1
#> <char> <num>
#> 1: c 1
#> 2: c 3
#> 3: c 6
#> 4: b 1
#> 5: b 3
#> 6: b 6
# .N can be different in i and j
DT[{cat(sprintf('in i, .N is %d\n', .N)); a < .N/2},
{cat(sprintf('in j, .N is %d\n', .N)); mean(a)}]
#> in i, .N is 9
#> in j, .N is 4
#> [1] 2.5
# .I can be different in j and by, enabling rowwise operations in by
DT[, .(.I, min(.SD[,-1]))]
#> I V2
#> <int> <num>
#> 1: 1 0.3333333
#> 2: 2 0.3333333
#> 3: 3 0.3333333
#> 4: 4 0.3333333
#> 5: 5 0.3333333
#> 6: 6 0.3333333
#> 7: 7 0.3333333
#> 8: 8 0.3333333
#> 9: 9 0.3333333
DT[, .(min(.SD[,-1])), by=.I]
#> I V1
#> <int> <num>
#> 1: 1 0.3333333
#> 2: 2 0.3333333
#> 3: 3 0.3333333
#> 4: 4 0.6666667
#> 5: 5 0.6666667
#> 6: 6 0.6666667
#> 7: 7 1.0000000
#> 8: 8 1.0000000
#> 9: 9 1.0000000