ios – 具有视频背景和绘画覆盖性能问题的全视频屏幕捕获

很长一段时间stackoverflow读者,第一次海报.

我正在尝试创建一个名为CloudWriter的iPad应用.应用程序的概念是绘制您在云中看到的形状.下载应用程序后,在启动CloudWriter后,将向用户呈现实时视频背景(来自后置摄像头),其上面有一个OpenGL绘图层.用户可以打开应用程序,将iPad指向天空中的云,并绘制他们在显示屏上看到的内容.

该应用程序的一个主要特征是用户记录会话期间显示器上发生的事情的视频屏幕捕获.实时视频输入和“绘图”视图将成为平面(合并)视频.

关于当前如何工作的一些假设和背景信息.

>使用Apples AVCamCaptureManager(AVCam示例项目的一部分)作为大部分相机相关代码的基础.
>使用AVCaptureSessionPresetMedium作为预设启动AVCamCapture会话.
>开始通过videoPreviewLayer将相机输出作为背景输出.
>使用openGL覆盖直播videoPreviewLayer,其视图允许“绘图”(手指绘画样式). “绘图”视图背景是[UIColor clearColor].

在这一点上,想法是用户可以将iPad3相机指向天空中的某些云,并绘制他们看到的形状.此功能完美无瑕.当我尝试对用户会话进行“平面”视频屏幕捕获时,我开始遇到性能问题.由此产生的“平面”视频会使摄像机输入与用户绘图实时重叠.

一个具有类似于我们正在寻找的功能的应用程序的一个很好的例子是Board Cam,可在App Store中找到.

要启动此过程,视图中始终会显示“记录”按钮.当用户点击记录按钮时,期望是在再次点击记录按钮之前,会话将被记录为“平面”视频屏幕捕获.

当用户点击“记录”按钮时,代码中会发生以下情况

> AVCaptureSessionPreset已从AVCaptureSessionPresetMedium更改为AVCaptureSessionPresetPhoto,允许访问

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection

> isRecording值设置为YEs.
> didOutputSampleBuffer开始获取数据并从当前视频缓冲区数据创建图像.它通过调用来完成此操作

- (UIImage *) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer

> self.currentImage设置为this

>应用程序根视图控制器开始覆盖drawRect以创建展平图像,用作最终视频中的单个帧.
>该帧被写入平面视频

为了创建一个平面图像,用作单独的框架,在根ViewController的drawRect@L_874_30@中,我们获取AVCamCaptureManager的didOutputSampleBuffer代码接收的最后一帧.那是在下面

- (void) drawRect:(CGRect)rect {


    NSDate* start = [NSDate date];
    CGContextRef context = [self createBitmapContextOfSize:self.frame.size];

    //not sure why this is necessary...image renders upside-down and mirrored
    CGAffineTransform flipVertical = CGAffineTransformMake(1,-1,self.frame.size.height);
    CGContextConcatCTM(context,flipVertical);

    if( isRecording)
        [[self.layer presentationLayer] renderInContext:context];

    CGImageRef cgImage = CGBitmapContextCreateImage(context);
    UIImage* BACkground = [UIImage imageWithCGImage: cgImage];
    CGImageRelease(cgImagE);

    UIImage *bottomImage = BACkground;


    if(((AVCamCaptureManager *)self.captureManager).currentImage != nil && isVideoBGActive )
    {

        UIImage *image = [((AVCamCaptureManager *)self.mainContentScreen.captureManager).currentImage retain];//[UIImage
        CGSize newSize = BACkground.size;
        UIGraphicsBeginImageContext( newSize );
        // Use exisTing opacity as is
        if( isRecording )
        {
            if( [self.mainContentScreen isVideoBGActive] && _recording)
            {
                [image drawInRect:CGRectMake(0,newSize.width,newSize.height)];
            }
            // Apply supplied opacity

            [bottomImage drawInRect:CGRectMake(0,newSize.height) blendMode:kCGBlendModeNormal alpha:1.0];

        }
        UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();

        UIGraphicsEndImageContext();

        self.currentScreen = newImage;



        [image release];
    }

    if (isRecording) {
        float millisElapsed = [[NSDate date] timeIntervalSinceDate:startedAt] * 1000.0;
        [self writeVideoFrameAtTime:CMTimeMake((int)R_918_11845@illisElapsed,1000)];
    }

    float processingSeconds = [[NSDate date] timeIntervalSinceDate:start];
    float delayRemaining = (1.0 / self.frameRatE) - processingSeconds;

    CGContextRelease(context);

    //redraw at the specified framerate
    [self performSELEctor:@SELEctor(setNeedsDisplay) withObject:nil afterDelay:delayRemaining > 0.0 ? delayRemaining : 0.01];  
}

createBitmapContextOfSize如下

- (CGContextRef) createBitmapContextOfSize:(CGSizE) size {
    CGContextRef    context = NULL;
    CGColorSpaceRef colorSpace = nil;
    int             bitmapByteCount;
    int             bitmapBytesPerRow;

    bitmapBytesPerRow   = (size.width * 4);
    bitmapByteCount     = (bitmapBytesPerRow * size.height);
    colorSpace = CGColorSpaceCreateDeviceRGB();
    if (bitmaPDAta != NULL) {
        free(bitmaPDAta);
    }
    bitmaPDAta = malloc( bitmapByteCount );
    if (bitmaPDAta == NULL) {
        fprintf (stderr,"Memory not allocated!");
        CGColorSpaceRelease( colorSpace );
        return NULL;
    }

    context = CGBitmapContextCreate (bitmaPDAta,size.width,size.height,8,// bits per component
                                     bitmapBytesPerRow,colorSpace,kCGImageAlphaPremultipliedFirst);

    CGContextSetAllowsAntialiasing(context,NO);
    if (context== NULL) {
        free (bitmaPDAta);
        fprintf (stderr,"Context not created!");
        CGColorSpaceRelease( colorSpace );
        return NULL;
    }

    //CGAffineTransform transform = CGAffineTransformIdentity;
    //transform = CGAffineTransformScale(transform,size.width * .25,size.height * .25);
    //CGAffineTransformScale(transform,1024,768);

    CGColorSpaceRelease( colorSpace );

    return context;
}

– (void)captureOutput：didOutputSampleBuffer fromConnection

// Delegate routIne that is called when a sample buffer was written
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection
{

    // Create a UIImage from the sample buffer data
    [self imageFromSampleBuffer:sampleBuffer];
}

– (UIImage *)imageFromSampleBuffer：(CMSampleBufferRef)sampleBuffer下面

// Create a UIImage from sample buffer data - modifed not to return a UIImage *,rather store it in self.currentImage
- (UIImage *) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer
{

    // unlock the memory,do other stuff,but don't forget:

    // Get a CMSampleBuffer's Core Video image buffer for the media data
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    // Lock the base address of the pixel buffer
    CVPixelBufferLockBaseAddress(imageBuffer,0);

   // uint8_t *tmp = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
    int bytes = CVPixelBufferGetBytesPerRow(imageBuffer); // determine number of bytes from height * bytes per row
    //void *baseAddress = malloc(bytes);
    size_t height = CVPixelBufferGetHeight(imageBuffer);     
    uint8_t *baseAddress = malloc( bytes * height );
    memcpy( baseAddress,CVPixelBufferGetBaseAddress(imageBuffer),bytes * height );
    size_t width = CVPixelBufferGetWidth(imageBuffer);

    // Create a device-dependent RGB color space
    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();

    // Create a bitmap graphics context with the sample buffer data
    CGContextRef context = CGBitmapContextCreate(baseAddress,width,height,bytes,kCGBitmapByteOrderDefault | kCGImageAlphaPremultipliedFirst);


    // CGContextScaleCTM(context,0.25,0.25); //scale down to size
    // Create a Quartz image from the pixel data in the bitmap graphics context
    CGImageRef quartzImage = CGBitmapContextCreateImage(context);
    // Unlock the pixel buffer
    CVPixelBufferUnlockBaseAddress(imageBuffer,0);

    // Free up the context and color space
    CGContextRelease(context);
    CGColorSpaceRelease(colorSpacE);
    free(baseAddress);

    self.currentImage = [UIImage imageWithCGImage:quartzImage scale:0.25 orientation:UIImageOrientationUp];

    // Release the Quartz image
    CGImageRelease(quartzImagE);


    return nil; 
}

最后,我使用writeVideoFrameAtTime：CMTimeMake将其写入磁盘,代码如下：

-(void) writeVideoFrameAtTime:(CMTimE)time {
    if (![videoWriterInput isReadyForMoreMediaData]) {
        NSLog(@"Not ready for video data");
    }
    else {
        @synchronized (self) {
            UIImage* newFrame = [self.currentScreen retain];
            CVPixelBufferRef pixelBuffer = NULL;
            CGImageRef cgImage = CGImageCreateCopy([newFrame CGImage]);
            CFDataRef image = CGDataProviderCopyData(CGImageGetDataProvider(cgImagE));

            if( image == nil )
            {
                [newFrame release];
                CVPixelBufferRelease( pixelBuffer );
                CGImageRelease(cgImagE);
                return;
            }

            int status = CVPixelBufferPoolCreatePixelBuffer(kcfAllocatorDefault,avAdaptor.pixelBufferPool,&pixelBuffer);
            if(status != 0){
                //Could not get a buffer from the pool
                NSLog(@"Error creaTing pixel buffer:  status=%d",status);
            }

            // set image data into pixel buffer
            CVPixelBufferLockBaseAddress( pixelBuffer,0 );
            uint8_t* destPixels = CVPixelBufferGetBaseAddress(pixelBuffer);
            CFDataGetBytes(image,CFRangeMake(0,CFDataGetLength(imagE)),destPixels);  //XXX:  will work if the pixel buffer is contiguous and has the same bytesPerRow as the input data

            if(status == 0){
                BOOL success = [avAdaptor appendPixelBuffer:pixelBuffer withPresentationTime:time];
                if (!success)
                    NSLog(@"Warning:  Unable to write buffer to video");
            }

            //clean up
            [newFrame release];
            CVPixelBufferUnlockBaseAddress( pixelBuffer,0 );
            CVPixelBufferRelease( pixelBuffer );
            CFRelease(imagE);
            CGImageRelease(cgImagE);
        }

    }

}

一旦isRecording设置为YES,iPad 3的性能就会从大约20FPS变为5FPs.使用Insturments,我能够看到以下代码块(来自drawRect :)导致性能下降到无法使用的级别.

if( _recording )
        {
            if( [self.mainContentScreen isVideoBGActive] && _recording)
            {
                [image drawInRect:CGRectMake(0,newSize.height) blendMode:kCGBlendModeNormal alpha:1.0];

        }

这是我的理解,因为我正在捕捉全屏,我们失去了“drawInRect”应该给予的所有好处.具体来说,我在谈论更快的重绘,因为理论上,我们只更新显示的一小部分(传入CGRect).再次,全屏捕捉,我不确定drawInRect可以提供几乎同样多的好处.

为了提高性能,我想如果我要缩小imageFromSampleBuffer提供的图像和绘图视图的当前上下文,我会看到帧速率的增加.不幸的是,CoreGrapics.Framework不是我过去曾经使用的东西,所以我不知道我能够有效地将性能调整到可接受的水平.

任何CoreGraphics Guru都有输入？

此外,ARC关闭了一些代码,分析仪显示一个泄漏,但我认为这是一个误报.

即将推出,CloudWriter™,天空是极限！