Changes between Version 36 and Version 37 of DataParallel/Replicate


Ignore:
Timestamp:
Aug 22, 2011 3:25:41 AM (3 years ago)
Author:
chak
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • DataParallel/Replicate

    v36 v37  
    185185=== Multiple levels of nesting (unconcat and concat) === 
    186186 
    187 TODO: What if we have segment descriptors on top of one with repeated segments? 
     187{{{ 
     188*Test> :type arrM6 
     189arrM6 :: PArray (PArray (PArray Int)) 
     190 
     191-- This array has a complex representation that includes multiple flat vectors (the PInt vectors). 
     192-- This is because it has been created by appending several other arrays together. 
     193 
     194*Test> pprv arrM6 
     195[[[7,8,9,10,11,12,13],[0],[1,2,3],[0]],[[0],[1,2,3]],[[0],[1,2,3],[5,6,7,8,9]],[[5,6,7,8,9]],[[1,2,3,4,5],[1,2,3],[7,8,9,10,11,12,13],[1,2,3]],[[5,6,7,8,9]]] 
     196 
     197*Test> pprp arrM6 
     198PArray  6 
     199   PNested 
     200       vsegids:    [0,1,2,3,4,5] 
     201       pseglens:   [4,2,3,1,4,1] 
     202       psegstarts: [0,4,6,9,10,14] 
     203       psegsrcs:   [0,0,0,0,0,0] 
     204       PNested 
     205           vsegids:    [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14] 
     206           pseglens:   [7,1,3,1,1,3,1,3,5,5,5,3,7,3,5] 
     207           psegstarts: [0,7,8,11,0,1,0,1,4,0,0,5,8,15,0] 
     208           psegsrcs:   [0,0,0,0,1,1,2,2,2,3,4,4,4,4,5] 
     209           PInt [7,8,9,10,11,12,13,0,1,2,3,0] 
     210           PInt [0,1,2,3] 
     211           PInt [0,1,2,3,5,6,7,8,9] 
     212           PInt [5,6,7,8,9] 
     213           PInt [1,2,3,4,5,1,2,3,7,8,9,10,11,12,13,1,2,3] 
     214           PInt [5,6,7,8,9] 
     215 
     216 
     217-- To pack the array we pack the vsegs, then drop the psegs that aren't referenced 
     218-- from any vseg. The pack operation only operates on the outer-most layer, 
     219-- so the inner segmentation isn't touched. 
     220 
     221*Test> pprv $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1 
     222[[[7,8,9,10,11,12,13],[0],[1,2,3],[0]], [[0],[1,2,3],[5,6,7,8,9]], [[5,6,7,8,9]]] 
     223 
     224*Test> pprp $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1 
     225PArray  0 
     226   PNested 
     227       vsegids:    [0,1,2] 
     228       pseglens:   [4,3,1] 
     229       psegstarts: [0,6,9] 
     230       psegsrcs:   [0,0,0] 
     231       PNested 
     232           vsegids:    [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14] 
     233           pseglens:   [7,1,3,1,1,3,1,3,5,5,5,3,7,3,5] 
     234           psegstarts: [0,7,8,11,0,1,0,1,4,0,0,5,8,15,0] 
     235           psegsrcs:   [0,0,0,0,1,1,2,2,2,3,4,4,4,4,5] 
     236           PInt [7,8,9,10,11,12,13,0,1,2,3,0] 
     237           PInt [0,1,2,3] 
     238           PInt [0,1,2,3,5,6,7,8,9] 
     239           PInt [5,6,7,8,9] 
     240           PInt [1,2,3,4,5,1,2,3,7,8,9,10,11,12,13,1,2,3] 
     241           PInt [5,6,7,8,9] 
     242 
     243 
     244-- Applying concat merges the two outer-most layers. For replicated arrays, this 
     245-- has the potential to copy data, but only at the outer-most level. 
     246 
     247*Test> pprv $ concatPA $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1 
     248[[7,8,9,10,11,12,13],[0],[1,2,3],[0],[0],[1,2,3],[5,6,7,8,9],[5,6,7,8,9]] 
     249 
     250*Test> pprp $ concatPA $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1 
     251PArray  8 
     252   PNested 
     253       vsegids:    [0,1,2,3,4,5,6,7] 
     254       pseglens:   [7,1,3,1,1,3,5,5] 
     255       psegstarts: [0,7,8,11,0,1,4,0] 
     256       psegsrcs:   [0,0,0,0,2,2,2,3] 
     257       PInt [7,8,9,10,11,12,13,0,1,2,3,0] 
     258       PInt [0,1,2,3] 
     259       PInt [0,1,2,3,5,6,7,8,9] 
     260       PInt [5,6,7,8,9] 
     261       PInt [1,2,3,4,5,1,2,3,7,8,9,10,11,12,13,1,2,3] 
     262       PInt [5,6,7,8,9] 
     263 
     264 
     265-- Concatenating the above array merges the segmentation with the flat arrays,  
     266-- producing a flat array. To put this another way, concat performs a 'gather' 
     267-- operation which forces out segmentation information by copying data.  
     268 
     269*Test> pprv $ concatPA $ concatPA $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1 
     270[7,8,9,10,11,12,13,0,1,2,3,0,0,1,2,3,5,6,7,8,9,5,6,7,8,9] 
     271 
     272*Test> pprp $ concatPA $ concatPA $ packByTagPA' arrM6 [1, 0, 1, 1, 0, 0] 1 
     273PArray  26 
     274   PInt [7,8,9,10,11,12,13,0,1,2,3,0,0,1,2,3,5,6,7,8,9,5,6,7,8,9] 
     275}}} 
    188276 
    189277== The bigger picture ==