Changes between Version 36 and Version 37 of DataParallel/Replicate


Ignore:
Timestamp:
Aug 22, 2011 3:25:41 AM (4 years ago)
Author:
chak
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • DataParallel/Replicate

    v36 v37  
    185185=== Multiple levels of nesting (unconcat and concat) ===
    186186
    187 TODO: What if we have segment descriptors on top of one with repeated segments?
     187{{{
     188*Test> :type arrM6
     189arrM6 :: PArray (PArray (PArray Int))
     190
     191-- This array has a complex representation that includes multiple flat vectors (the PInt vectors).
     192-- This is because it has been created by appending several other arrays together.
     193
     194*Test> pprv arrM6
     195[[[7,8,9,10,11,12,13],[0],[1,2,3],[0]],[[0],[1,2,3]],[[0],[1,2,3],[5,6,7,8,9]],[[5,6,7,8,9]],[[1,2,3,4,5],[1,2,3],[7,8,9,10,11,12,13],[1,2,3]],[[5,6,7,8,9]]]
     196
     197*Test> pprp arrM6
     198PArray  6
     199   PNested
     200       vsegids:    [0,1,2,3,4,5]
     201       pseglens:   [4,2,3,1,4,1]
     202       psegstarts: [0,4,6,9,10,14]
     203       psegsrcs:   [0,0,0,0,0,0]
     204       PNested
     205           vsegids:    [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14]
     206           pseglens:   [7,1,3,1,1,3,1,3,5,5,5,3,7,3,5]
     207           psegstarts: [0,7,8,11,0,1,0,1,4,0,0,5,8,15,0]
     208           psegsrcs:   [0,0,0,0,1,1,2,2,2,3,4,4,4,4,5]
     209           PInt [7,8,9,10,11,12,13,0,1,2,3,0]
     210           PInt [0,1,2,3]
     211           PInt [0,1,2,3,5,6,7,8,9]
     212           PInt [5,6,7,8,9]
     213           PInt [1,2,3,4,5,1,2,3,7,8,9,10,11,12,13,1,2,3]
     214           PInt [5,6,7,8,9]
     215
     216
     217-- To pack the array we pack the vsegs, then drop the psegs that aren't referenced
     218-- from any vseg. The pack operation only operates on the outer-most layer,
     219-- so the inner segmentation isn't touched.
     220
     221*Test> pprv $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1
     222[[[7,8,9,10,11,12,13],[0],[1,2,3],[0]], [[0],[1,2,3],[5,6,7,8,9]], [[5,6,7,8,9]]]
     223
     224*Test> pprp $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1
     225PArray  0
     226   PNested
     227       vsegids:    [0,1,2]
     228       pseglens:   [4,3,1]
     229       psegstarts: [0,6,9]
     230       psegsrcs:   [0,0,0]
     231       PNested
     232           vsegids:    [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14]
     233           pseglens:   [7,1,3,1,1,3,1,3,5,5,5,3,7,3,5]
     234           psegstarts: [0,7,8,11,0,1,0,1,4,0,0,5,8,15,0]
     235           psegsrcs:   [0,0,0,0,1,1,2,2,2,3,4,4,4,4,5]
     236           PInt [7,8,9,10,11,12,13,0,1,2,3,0]
     237           PInt [0,1,2,3]
     238           PInt [0,1,2,3,5,6,7,8,9]
     239           PInt [5,6,7,8,9]
     240           PInt [1,2,3,4,5,1,2,3,7,8,9,10,11,12,13,1,2,3]
     241           PInt [5,6,7,8,9]
     242
     243
     244-- Applying concat merges the two outer-most layers. For replicated arrays, this
     245-- has the potential to copy data, but only at the outer-most level.
     246
     247*Test> pprv $ concatPA $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1
     248[[7,8,9,10,11,12,13],[0],[1,2,3],[0],[0],[1,2,3],[5,6,7,8,9],[5,6,7,8,9]]
     249
     250*Test> pprp $ concatPA $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1
     251PArray  8
     252   PNested
     253       vsegids:    [0,1,2,3,4,5,6,7]
     254       pseglens:   [7,1,3,1,1,3,5,5]
     255       psegstarts: [0,7,8,11,0,1,4,0]
     256       psegsrcs:   [0,0,0,0,2,2,2,3]
     257       PInt [7,8,9,10,11,12,13,0,1,2,3,0]
     258       PInt [0,1,2,3]
     259       PInt [0,1,2,3,5,6,7,8,9]
     260       PInt [5,6,7,8,9]
     261       PInt [1,2,3,4,5,1,2,3,7,8,9,10,11,12,13,1,2,3]
     262       PInt [5,6,7,8,9]
     263
     264
     265-- Concatenating the above array merges the segmentation with the flat arrays,
     266-- producing a flat array. To put this another way, concat performs a 'gather'
     267-- operation which forces out segmentation information by copying data.
     268
     269*Test> pprv $ concatPA $ concatPA $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1
     270[7,8,9,10,11,12,13,0,1,2,3,0,0,1,2,3,5,6,7,8,9,5,6,7,8,9]
     271
     272*Test> pprp $ concatPA $ concatPA $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1
     273PArray  26
     274   PInt [7,8,9,10,11,12,13,0,1,2,3,0,0,1,2,3,5,6,7,8,9,5,6,7,8,9]
     275}}}
    188276
    189277== The bigger picture ==