UI 在 CI 上测试 Flaky

大佬教程收集整理的这篇文章主要介绍了UI 在 CI 上测试 Flaky，大佬教程大佬觉得挺不错的，现在分享给大家，也给大家做个参考。

如何解决UI 在 CI 上测试 Flaky？

开发过程中遇到UI 在 CI 上测试 Flaky的问题如何解决？下面主要结合日常开发的经验，给出你关于UI 在 CI 上测试 Flaky的解决方法建议，希望对你解决UI 在 CI 上测试 Flaky有所启发或帮助；

更新：

VNC 进入构建机器（不做任何更改）以某种方式修复了这个......奇怪

---

背景

我们使用 Mac EC2 实例 CI 集群来运行 UI 测试。集群设置为每晚回收。这意味着新集群是在早上配置的。我们注意到测试在一天中变得越来越不稳定。

• 构建 312 是早上的第一个构建。
• 红色是失败的测试，橙色是静音测试，黄色是片状测试，灰色是未知状态
• 提交只是触发构建的空白自动提交。测试内容相同。
• 注意：Bluepill 配置为每台机器运行一个模拟器（无并发）

日志

片状测试或失败通常是由于“等待某些资源空闲”并达到超时，或某些网络请求完成。失败或不稳定的测试也有一个共同点，即它们通常与视频有关

这里有几个示例日志

等待应用空闲

Timed out waiTing for app to IDle. 
The following IDling resources are busy.
1. GREYAppStateTracker:
WaiTing for network requests to finish. By default,EarlGrey tracks all network requests. To change this behavior,refer to GREYConfiguration.
<__NSCFLocalDataTask:0x7fcfcfd87450,URL:"https://v.myorgimg.com/vIDeos/mc/hls/e7/01/ca/e701caeBAC12684056eb699f316fc411_360w.m3u8"> => WaiTing for network requests to finish. By default,refer to GREYConfiguration.
(
    0   MyorgDevelopmentEG2             0x000000010ebf3205 -[GREYAppStateTrackerObject setState:] + 69
    1   MyorgDevelopmentEG2             0x000000010ebf22b3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDealLOCATIOnTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
    2   MyorgDevelopmentEG2             0x000000010ebf18f0 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
    3   MyorgDevelopmentEG2             0x000000010ebf1e34 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDealLOCATIOnTracker:orExternalAppStateTrackerObject:] + 1076
    4   MyorgDevelopmentEG2             0x000000010ebf0ac6 -[GREYAppStateTracker trackState:forObject:] + 102
    5   MyorgDevelopmentEG2             0x000000010ebca867 -[__NSCFLocalDataTask_GREYApp grey_track] + 135
    6   MyorgDevelopmentEG2             0x000000010ebcac55 -[__NSCFLocalDataTask_GREYApp greyswizzled_resume] + 181
    7   MediaToolBox                        0x00007fff2b617285 fighttprequestSetupNSURLSessionTask + 1380
    8   MediaToolBox                        0x00007fff2b616bb6 _fighttprequestCreateWithNSURLSession + 2923
    9   MediaToolBox                        0x00007fff2b64aaaf fighttprequestSessionNSCreatehttprequest + 199
    10  MediaToolBox                        0x00007fff2b71f5b1 segPumpCreatehttprequest + 944
    11  MediaToolBox                        0x00007fff2b721de6 segPumpSendindexfilerequest + 1703
    12  MediaToolBox                        0x00007fff2b7431d5 segPumprequesTindexForStream + 753
    13  MediaToolBox                        0x00007fff2b7421ee segPumpSetAlternateForStream + 3145
    14  MediaToolBox                        0x00007fff2b73e9e5 segPumpSetCurrentAlternate + 9898
    15  MediaToolBox                        0x00007fff2b791d6a ProduceStreamingAssetProperty + 661
    16  MediaToolBox                        0x00007fff2b78e07c URLAssetPropertyWorkFunction + 872
    17  libdispatch.dylib                   0x00007fff201078df _dispatch_clIEnt_callout + 8
    18  libdispatch.dylib                   0x00007fff2010de15 _dispatch_lane_serial_drain + 715
    19  libdispatch.dylib                   0x00007fff2010e9c3 _dispatch_lane_invoke + 455
    20  libdispatch.dylib                   0x00007fff20117a8e _dispatch_root_queue_drain + 350
    21  libdispatch.dylib                   0x00007fff2011786c _dispatch_worker_thread + 222
    22  libsystem_pthread.dylib             0x00007fff60342950 _pthread_start + 224
    23  libsystem_pthread.dylib             0x00007fff6033e47b thread_start + 15
)

等待网络请求完成

9. GREYAppStateTracker:
WaiTing for network requests to finish. By default,refer to GREYConfiguration.
WaiTing for UIVIEw's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
<ASCollectionVIEw: 0x7feca21ad000> => WaiTing for UIVIEw's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
(
    0   MyorgDevelopmentEG2             0x000000010bb2e335 -[GREYAppStateTrackerObject setState:] + 69
    1   MyorgDevelopmentEG2             0x000000010bb2d3e3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDealLOCATIOnTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
    2   MyorgDevelopmentEG2             0x000000010bb2ca20 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
    3   MyorgDevelopmentEG2             0x000000010bb2cf64 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDealLOCATIOnTracker:orExternalAppStateTrackerObject:] + 1076
    4   MyorgDevelopmentEG2             0x000000010bb2bbf6 -[GREYAppStateTracker trackState:forObject:] + 102
    5   MyorgDevelopmentEG2             0x000000010bb02b0d -[UIVIEw(GREYApp) greyswizzled_setNeedsLayout] + 93
    6   UIKitCore                           0x00007fff24ba5675 -[UIScrollVIEw setNeedsLayout] + 79
    7   UIKitCore                           0x00007fff23dc0390 -[UICollectionVIEw _updateAnimationDIDStop:finished:co@R_262_10443@t:] + 2236
    8   UIKitCore                           0x00007fff23dbf9e4 __102-[UICollectionVIEw _updateWithItems:tentativelyForReordering:propertyAnimator:collectionVIEwAnimator:]_block_invoke.2110 + 77
    9   MyorgDevelopmentEG2             0x000000010bb2fdef __60-[GREYdispatchQueueTracker grey_dispatchAsyncCallWithBlock:]_block_invoke + 47
    10  libdispatch.dylib                   0x00007fff2010670d _dispatch_call_block_and_release + 12
    11  libdispatch.dylib                   0x00007fff201078df _dispatch_clIEnt_callout + 8
    12  libdispatch.dylib                   0x00007fff20114a27 _dispatch_main_queue_callBACk_4CF + 1045
    13  CoreFoundation                      0x00007fff203908f8 __CFRUNLOOP_IS_serviciNG_THE_MAIN_disPATCH_QUEUE__ + 9
    14  CoreFoundation                      0x00007fff2038b169 __CFRunLoopRun + 2781
    15  CoreFoundation                      0x00007fff2038a1a7 CFRunLoopRunspecific + 567
    16  GraphiCSservices                    0x00007fff2b874d85 GSEventRunModal + 139
    17  UIKitCore                           0x00007fff246c14df -[UIApplication _run] + 912
    18  UIKitCore                           0x00007fff246c639c UIApplicationMain + 101
    19  MyorgDevelopmentEG2             0x0000000108f175de main + 190
    20  libdyld.dylib                       0x00007fff2025abbd start + 1
    21  ???                                 0x0000000000000005 0x0 + 5
)
<_ASdisplayLayer: 0x600007996580> => WaiTing for UIVIEw's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.

假设

这让我认为有一些孤立的子进程或守护进程仍然占有资源。或者没有正确清理工件。

我尝试过的东西

• 关闭、擦除、删除构建之间的所有模拟器（无帮助）

    xcrun simctl shutdown all 
    xcrun simctl erase all
    xcrun simctl delete all
    osascript -e 'tell application "iOS Simulator" to quit'
    osascript -e 'tell application "Simulator" to quit'

• 在构建之间终止 CoreSimulator 相关服务（无帮助）

    launchctl remove com.apple.CoreSimulator.CoreSimulatorservice || true
    launchctl remove com.apple.CoreSimulator.SimLaunchHost-x86 || true
    launchctl remove com.apple.CoreSimulator.SimulatorTrampoline || true
    launchctl remove com.apple.CoreSimulator.SimLaunchHost-arm64 || true

• 在构建之间删除模拟器设备、日志和缓存。这实际上有点帮助。片状最初消失了，但在几次构建后很快又回来了，而机器仍在构建之间执行清洁......不确定为什么。

    echo "delete all simulator devices,cache,and temp"
    rm -rf ~/library/Developer/CoreSimulator

    echo "delete Simulator Logs"
    rm -rf ~/library/Logs/CoreSimulator

• 在 EarlGrey 上增加 Animation Timeout，Interaction Timeout
• Blacklist the URL 请求挂起。这确实通过忽略症状而不修复实际问题来提高测试套件的稳定性，所以我希望我可以避免这种情况。

还有其他人遇到同样的问题吗？感谢有人可以对此有所了解或提供尝试建议。

注意：我们使用 Bluepill 和 EarlGrey2 在 buildkite CI 管道上运行 UI 测试（iOS 14.5、Xcode 12.5）。

解决方法

大佬总结

以上是大佬教程为你收集整理的UI 在 CI 上测试 Flaky全部内容，希望文章能够帮你解决UI 在 CI 上测试 Flaky所遇到的程序开发问题。

如果觉得大佬教程网站内容还不错，欢迎将大佬教程推荐给程序员好友。

本图文内容来源于网友网络收集整理提供，作为学习参考使用，版权属于原作者。
如您有任何意见或建议可联系处理。小编QQ：384754419，请注明来意。