#Design Process

https://www.youtube.com/watch?v=g2vlqhefADk

1. 搜尋符合現有task的architecture

• transfer learning: using model that pre-trained on large open dataset and fine-tune
• 如果需要special case, things that not showed up before, THEN

2. 搜尋好用的design patterns

(1) for CV

• efficient model architectures

  • eg. EfficientNet, ShuffleNet, MobileNet...
  • seperable depthwise conv
    • 減少計算量、參數量
    • 最後使用1x1conv來學習跨通道特徵
    • 舉例：input(32x32x256), output(32,32,128)
      • 原本要有128個3x3x256kernel做計算
        • 參數量(少算bias):3x3x256x128
        • 計算量(少算加法與bias):32x32x3x3x256x128
      • 現在只要先用256個3x3x1kernel分別對256channel作卷積後(不sum)輸出256張map, 再用128個1x1x256的conv來獲取跨通道特徵, 輸出128張feature map
        • 參數量(少算bias):256x3x3+128x1x1x256
        • 計算量(少算加法與bias):32x32x3x3

  Some Practical Guidelines in paper: ShuffleNetV2

  1. 在一個conv layer的兩端用equal channel width的feature map可以minimize memory access cost
  • not to change input/output channels of a layer too frequently
  • 當然也有打破規則的, eg. SqueezeNet, MobileNetV2
  1. extremely narrow bottlenecks hurt training stability
  • bottleneck: layers having very few filters(<=8), 通常是指為減少計算量而使用1x1conv先來降維的中間層
  • a few dead neurons and the entire model collapses
  • using linear activation instead of ReLU after previous conv layer
  1. network fragmentation reduces the degree of parallelism
  • using a few large operations is better than a lot of small ones
  • since the later one is not very GPU-friendly and introduce extra overhead(額外付出的時間)
  1. point-wise operations eg.ReLU and 1x1Conv are not free
  • Memory access cost is non-negligibal although small number of fp operations

• inception network

  • 獲得不同空間尺度特徵(4條路線), 並在加深的過程降低計算量
    • 不同空間尺度： 1x1, 3x3, 5x5, 3x3 max pooling
    • 降低計算量：直接3x3, 5x5計算量太大, 在這之前先經過1x1到Channel較低的feature map再3x3, 5x5
    • 四條路線的最後H, W都一樣(使用padding), 而C就是四個concat起來, 所以size = (H,W,C1+C2+C3+C4)
  • 1x1的功能：降維(形成bottleneck)並在過程中跨通道學習特徵、降低運算量及參數量
    • 降維(做Cin*Cout的線性映射)
      1. 假設現在input(H,W,C)為10x10x16
      2. 看著每個channel中位置(5,5)的點
      3. 1x1的一個核(1x1x16)相當於對(5,5,1),(5,5,2)...(5,5,16)做線性映射運算(對每個點乘以各自的參數後相加,每個參數其實就是1x1核的其中一個切片) $in\ output\ feature\ map_1:\new\ (5,5,1)=w_{1,1}(5,5,1)+w_{1,2}(5,5,2)...+w_{1,16}(5,5,16)$
      4. 若output channel為10, 則 $new\ (5,5,2)=w_{2,1}(5,5,1)+w_{2,2}(5,5,2)...+w_{1,16}(2,5,16)\...\new\ (5,5,10)=w_{10,1}(5,5,1)+w_{10,2}(5,5,2)...+w_{10,16}(2,5,16)$
      5. 其實就是16 to 10 neurons 的fc (16*10)
      6. 類推到其他所有點(1,1)~(10,10)
    • a channel-wise dense layer that learn cross channel features

• residual block

  • skip connections between many layers so that adding more layers won't worsen performance
  • create specail path for the gradient to flow back to ealier layer more easily
  • 通常都用add/concat來連接
  • 應用
    1. in fully conv networks (連接於對稱的位置)
    • to combine information from deep and shallow layers to produce pixel-wise segmentation maps.
    • 可以幫助後面的layer還原原圖片物件之位置, 大小等空間資訊 help recover fine spatial information discarded by coarse layers while preserving coarse strucures
    1. in blocks (連接一個block前後)
    • to reuse features from previous layers

• attention

(2) for NLP

1. attention, Transfomer

3. NAS

• 現階段可能還是輸hand-made, 有論文研究NAS實際上還是比Efficient Model還慢很多就算fp operations比較少, 可能因為NAS容易產生太多network fragmentations
• 權重無關網路搜索

4. Custom Model

Conv Model

(1) How to choose the number of layers and units?