Building Images for multiple monitors in windows

Just a Penguin - Slide

Not familiar with Golang? Check out A Tour of Go!
Overview

During the slide project one of our main focuses was allowing users full control over how images were displayed on their monitors. These days a large percentage of users utilise multiple monitors, so we set about implementing a robust set of tools for managing backgrounds in multi-monitor setups on Windows, Mac OS and Ubuntu. This meant allowing resolution and aspect ratio filtering on each monitor and proper image scaling options for each monitor.

It would be reasonable to assume that this would be simple to implement on windows. You can already set different images as the desktop background on different monitors on windows [1], so surely a separate application can hook into and use this system? No such luck. In fact, out of all of the operating systems Windows was the most difficult to work with in this case.

In this article I will go through our methodology for solving this problem, with full go code examples and visual representations.



So How Does it Work?

Say we have two 1080p monitors and two images that we want to place on each monitor. One image is 1080p and the other is 1440p. From the windows registry we can get the width, height and x, y positions of each monitor, with 0, 0 being the top left corner of the primary monitor and the positions of any others relative to that. Fig. 0 shows a visual representation of the data we would receive from the registry for our monitor setup, as well as the resolution of the images in comparison to the monitors.

Fig. 0. Monitor Sizes and Positions

Well, -1920 as the x co-ordinate of the left monitor is actually wishful thinking. Unfortunately the position integer in windows is incapable of being negative, so the real value would be an overflow from the integers maximum possible value. Because of this we use a bit of maths to correct the x co-ordinate of the furthest left monitor to 0 (and correct the co-ordinates of any other monitors accordingly). The same calculation is required in the y direction for any monitors that may be above the primary monitor.

Now that we know the monitor layout we can create a canvas the size of both of them and map the images to the canvas in the correct positions. This is simple when the dimensions of the image and the screen are identical, but becomes more difficult in the case of image 0, which is larger than monitor 0. There are multiple methods of fitting an image to a screen which are used by the windows wallpaper dialog; fit, fill, stretch etc. We will look at implementing these different methods as options for scaling image 0 to fit screen 0.



Getting Monitor Information

Windows stores information about the current monitor setup in the registry [2]. Each time a new setup is detected a new key is created, with keys containing variables for each monitor. We get the most recent monitor setup by finding the key with the most recent timestamp variable (Fig. 1).

// GetMonitors returns size and position information about all known monitors.
func GetMonitors() ([]*Monitor, error) {
	//Open the registry key that stores monitor information
	//0x20019 is KEY_READ access, combining STANDARD_RIGHTS_READ, KEY_QUERY_VALUE, KEY_ENUMERATE_SUB_KEYS, and KEY_NOTIFY values.
	k, err := registry.OpenKey(registry.LOCAL_MACHINE, `SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration`, 0x20019)
	if err != nil {
		return nil, errors.Wrap(err, "open registry key failed")
	}
	defer k.Close()
	subKeys, err := getSubKeys(k)
	if err != nil {
		return nil, errors.Wrap(err, "get registry subkeys failed")
	}
	var setupSubKeys []*setupSubKey
	//For the range of subkeys get the timestamp and ID, and save in a slice of monitorSubKey structs
	//Could move the contents of this for loop to a function, allowing for defer l.Close()
	for _, subKey := range subKeys {
		l, err := registry.OpenKey(registry.LOCAL_MACHINE, fmt.Sprintf(`SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration\%s`, subKey), 0x20019)
		if err != nil {
			return nil, errors.Wrap(err, "open registry key failed")
		}
		timeStamp, _, err := l.GetIntegerValue("Timestamp")
		if err != nil {
			return nil, errors.Wrap(err, "load registry value failed")
		}
		setupSubKeys = append(setupSubKeys, &setupSubKey{
			timeStamp: timeStamp,
			subKey:    subKey,
		})
		l.Close()
	}

	//Find the member of the slice with the most recent timestamp and get its key
	sort.Sort(ByTime(setupSubKeys))

	monitorKey := setupSubKeys[len(setupSubKeys)-1]

Fig. 1. Get Most Recent Monitor Setup

Now that we have the most recent monitor setup we need to open each monitor sub key and save the information for each monitor. To do this we create a struct for each present monitor that contains the relevant monitor information and a unique identifier and append it to a slice of structs.

//Find the member of the slice with the most recent timestamp and get its key
	sort.Sort(ByTime(setupSubKeys))

	monitorKey := setupSubKeys[len(setupSubKeys)-1]

	m, err := registry.OpenKey(registry.LOCAL_MACHINE, fmt.Sprintf(`SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration\%s`, monitorKey.subKey), 0x20019)
	if err != nil {
		return nil, errors.Wrap(err, "open registry key failed")
	}

	// The getSubKeys function returns the names of all subkeys in m, each of these is a monitor
	monitorSubKeys, err := getSubKeys(m)
	if err != nil {
		return nil, errors.Wrap(err, "get registry subkeys failed")
	}

	var monitors []*Monitor

	//For each monitor subkey, get the position and dimensions
	for _, monitorSubKey := range monitorSubKeys {
		o, err := registry.OpenKey(registry.LOCAL_MACHINE, fmt.Sprintf(`SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration\%s\%s`, monitorKey.subKey, monitorSubKey), 0x20019)
		if err != nil {
			return nil, errors.Wrap(err, "open registry key failed")
		}

		posX64, _, err := o.GetIntegerValue("Position.cx")
		if err != nil {
			return nil, errors.Wrap(err, "load registry value failed")
		}
		posX := int(posX64)

		posY64, _, err := o.GetIntegerValue("Position.cy")
		if err != nil {
			return nil, errors.Wrap(err, "load registry value failed")
		}
		posY := int(posY64)

		width64, _, err := o.GetIntegerValue("PrimSurfSize.cx")
		if err != nil {
			return nil, errors.Wrap(err, "load registry value failed")
		}
		width := int(width64)

		height64, _, err := o.GetIntegerValue("PrimSurfSize.cy")
		if err != nil {
			return nil, errors.Wrap(err, "load registry value failed")
		}
		height := int(height64)

		//Find the greatest common divisor
		result := gcd(width, height)
		//Get the aspect ratio of the monitor
		aspectX := width / result
		aspectY := height / result

		//Append the monitor data to the monitor struct slice
		monitors = append(monitors, &Monitor{
			PosX:    posX,
			PosY:    posY,
			AspectX: aspectX,
			AspectY: aspectY,
			Width:   width,
			Height:  height,
			ID:      "",
		})
	}

Fig. 2. Get Monitor Info

In Fig. 3 we correct any overflowed values for monitor position in x and y to give the correct negative position in lines 99-112. Then in lines 114-124 we find the smallest (or most negative) x and y positions. These will always be less than or equal to zero, so we then correct all of the monitor position values so that the co-ordinates are always from a (0,0) origin. For example in a two monitor setup as shown in Fig. 0 the corrected co-ordinates would be (0,0), (1920, 0). We do this by decrementing the position values by the value of the most negative positions in lines 126-128. Then we create a unique ID for each monitor so we can identify them easily, and assign a number to each monitor before returning the struct.

		//Append the monitor data to the monitor struct slice
		monitors = append(monitors, &Monitor{
			PosX:    posX,
			PosY:    posY,
			AspectX: aspectX,
			AspectY: aspectY,
			Width:   width,
			Height:  height,
			ID:      "",
		})
	}

	for _, monitor := range monitors {
		//If position has wrapped around (if a monitor is on the left or above) then find the correct (negative!) number
		if int64(monitor.PosX) > winOverflowLimit {
			//Gives the positive value of position
			newPos := -(winOverflowAdjust - int64(monitor.PosX) + 1)
			monitor.PosX = int(newPos)
		}

		if int64(monitor.PosY) > winOverflowLimit {
			newPos := -(winOverflowAdjust - int64(monitor.PosY) + 1)

			monitor.PosY = int(newPos)
		}
	}

	minPosX, minPosY := monitors[0].PosX, monitors[0].PosY

	for _, monitor := range monitors {
		if monitor.PosX <= minPosX {
			minPosX = monitor.PosX
		}

		if monitor.PosY <= minPosY {
			minPosY = monitor.PosY
		}
	}

	for index, monitor := range monitors {
		monitor.PosX -= minPosX
		monitor.PosY -= minPosY

		//Create a unique id for each monitor using dimensions and position
		monitor.ID = MonitorID(fmt.Sprintf("%d-%d-%d-%d", monitor.Width, monitor.Height, monitor.PosX, monitor.PosY))
		monitor.Num = index
	}

	return monitors, nil
}

Fig. 3. Correct Monitor Info



Creating the Image Canvas

Now that we have the current monitor information we can use that information to choose images and apply them to our monitor canvas in the correct positions with the correct scaling.

Fig. 4 shows the function SetMultiWall which can be called after the monitor information is known. It takes three arguments; a slice of monitor structs, a slice of file paths that point to images, one for each monitor, and a slice of display modes, one for each monitor. In Slide the images are collected based on user inputted filters, and the mode can be set by the user, although we will not go into these systems here.

SetMultiWall acts as middleware, first we ensure that the Windows display mode is set to tiled using the SetDisplayMode function. This function accesses the registry and sets the mode there. This is necessary as tiled mode will not try to modify our canvas of images in any way, whereas the other Windows display modes may.

Then we use the function makeMultiWall to build our canvas of images, this function is explored in detail in Fig. 5. Finally we call SetFromFile, which sets the windows desktop background to be the canvas of images returned by makeMultiWall.

// SetMultiWall sets each given image on the corresponding Monitor.
// If a certain wallpaper cannot be applied, an error is returned with the corresponding index
// so that it can be dealt with (e.g. removed and reattempted)
func SetMultiWall(monitors []*Monitor, images []string, modes []string) (int, error) {
	err := SetDisplayMode(Tile)

	if err != nil {
		return -1, err
	}

	multiWallPath, failedIndex, err := makeMultiWall(monitors, images, modes)

	if err != nil {
		return failedIndex, err
	}

	return -1, SetFromFile(multiWallPath)
}

Fig. 4. SetMultiWall

Fig. 5 shows the function makeMultiWall, which creates the canvas of images to be used as the desktop background. It takes a slice of pointers to monitor structs, a slice of images and a slice of modes. It returns a string that contains the file path of the canvas of images, an integer that can be used to identify which monitor caused an error (if applicable) and an error.

First, in lines 4-13 we find the total width and height of the canvas by finding the largest result of monitor position and size in x and y. Then we create a new RGBA image based on a rectangle the size of our canvas in lines 16-17. Then we need to process the image for each monitor. This is handled by the processMonitorImage function which is covered in Fig. 6. This function is called in a loop for each monitor in lines 19-25. Finally the RGBA canvas of images is encoded into a jpeg and saved in lines 28-56. First the users cache directory is found, then filepath.Join adds the name of the canvas of images to the cache directory, this path is then created using os.Create and the RGBA image is encoded to it using jpeg.Encode. JPEG is used here as Windows has troubles setting PNG images with large file sizes as desktop backgrounds, otherwise PNG would be preferable.

func makeMultiWall(monitors []*Monitor, images []string, modes []string) (string, int, error) {
	totalWidth, totalHeight := 0, 0

	for _, monitor := range monitors {
		newWidth := monitor.PosX + monitor.Width
		if newWidth >= totalWidth {
			totalWidth = newWidth
		}
		newHeight := monitor.PosY + monitor.Height
		if newHeight >= totalHeight {
			totalHeight = newHeight
		}
	}

	// Create a rectangle to hold all the images
	r := image.Rectangle{Min: image.Point{X: 0, Y: 0}, Max: image.Point{X: totalWidth, Y: totalHeight}}
	rgba := image.NewRGBA(r)

	for index, monitor := range monitors {
		err := processMonitorImage(monitor, images[index], modes[index], rgba)

		if err != nil {
			return "", index, err
		}
	}

	// Don't allow another call of the function to try to open tmp.jpg whilst it is being used
	tmpPathMutex.Lock()
	defer tmpPathMutex.Unlock()

	cacheDir, err := getCacheDir()

	if err != nil {
		return "", -1, errors.Wrapf(err, "unable to get cache dir")
	}

	tmpPath := filepath.Join(cacheDir, "multi-wall.jpg")

	outImage, err := os.Create(tmpPath)

	if err != nil {
		return "", -1, errors.Wrapf(err, "unable to create temp path at: %s", tmpPath)
	}

	defer outImage.Close()

	output := bufio.NewWriter(outImage)

	err = jpeg.Encode(output, rgba, &jpeg.Options{Quality: 100})

	if err != nil {
		return "", -1, errors.Wrap(err, "unable to encode multi-monitor image")
	}

	return tmpPath, -1, err
}

Fig. 5. MakeMultiWall

In Fig. 6 the first part of the function processMonitorImage is shown. This function is responsible for taking an image for a monitor, resizing and cropping it based on the selected mode then printing it onto the canvas of images in the correct position. It takes a pointer to a monitor struct, a path to an image, a mode string and an RGBA image as arguments.

In lines 3-17 we open and decode the image. In lines 20-36 three if statements resize the image according to the selected mode. The different image display modes are visualised in Fig. 8. In order to resize images the Resize package is used.

func processMonitorImage(monitor *Monitor, imagePath string, mode string, drawImage draw.Image) error {
	// Open the images, these should be loaded from the database based on screen resolutions found
	monitorImage, err := os.Open(imagePath)

	if err != nil {
		return errors.Wrapf(err, "unable to open image file: %s", monitorImage)
	}

	defer monitorImage.Close()

	input := bufio.NewReader(monitorImage)

	decodedImage, _, err := image.Decode(input)

	if err != nil {
		return ErrDecodeImage
	}

	// Resize the image to the dimensions of the monitor
	if mode == Stretch {
		decodedImage = resize.Resize(uint(monitor.Width), uint(monitor.Height), decodedImage, resize.Lanczos3)
	}

	// Resize the image to the dimensions of the monitor whilst maintaining aspect ratio
	if mode == Fit || mode == FitCenter {
		decodedImage = resize.Resize(uint(monitor.Width), 0, decodedImage, resize.Lanczos3)
	}

	// Resize the image to fill the monitor whilst maintaining aspect ratio
	if mode == Fill || mode == FillCenter {
		decodedImage = resize.Resize(uint(monitor.Width), 0, decodedImage, resize.Lanczos3)

		if decodedImage.Bounds().Max.Y < monitor.Height {
			decodedImage = resize.Resize(0, uint(monitor.Height), decodedImage, resize.Lanczos3)
		}
	}

	// Set the position of the image (top left)
	x := monitor.PosX
	y := monitor.PosY

Fig. 6. ProcessMonitorImage - Open and Resizing

In Fig. 7 the second part of the function processMonitorImage is shown. In lines 39-43 we set variables x and y to the x and y co-ordinates of the monitor that the image is to be displayed on and set the canvas width and height to the width and height of the monitor. If centered mode is not enabled then these sizes will be the resolution that the image is drawn as and these co-ordinates will be the point on the canvas that the image is drawn from.

If centered mode is enabled (or centered fit/fill) then the x and y co-ordinates need to be corrected to center the image, and if the image is larger in resolution than the monitor any excess needs to be cropped out. the Cutter library is used for cropping. This is handled in lines 46-80. For each axis we check if the image resolution is larger than the monitor resolution, if it isn't then we simply move the image to the center by calculating half of the difference between the size of the monitor and the image and shifting the image on the canvas by that amount. If the image resolution is larger then we shift the image by the same difference, correct the canvas size to the image size minus half of the difference, then crop the repositioned image to the size of the monitor.

Finally in lines 82-92 we draw the image to the main canvas of images with the correct origin point and dimensions.

	// Set the position of the image (top left)
	x := monitor.PosX
	y := monitor.PosY

	canvasWidth := monitor.Width
	canvasHeight := monitor.Height

	// Keep the image size but position it in the center of the canvas
	if mode == Center || mode == FitCenter || mode == FillCenter {
		if decodedImage.Bounds().Max.X < monitor.Width {
			x = monitor.Width - (((monitor.Width) - (decodedImage.Bounds().Max.X)) / 2) - decodedImage.Bounds().Max.X + monitor.PosX
		}

		if decodedImage.Bounds().Max.X > monitor.Width {
			x = monitor.Width - (((decodedImage.Bounds().Max.X) - (monitor.Width)) / 2) - monitor.Width + monitor.PosX
			canvasWidth = decodedImage.Bounds().Max.X - (((decodedImage.Bounds().Max.X) - (monitor.Width)) / 2)
			decodedImage, err = cutter.Crop(decodedImage, cutter.Config{
				Width:  monitor.Width,
				Height: decodedImage.Bounds().Max.Y,
				Mode:   cutter.Centered,
			})
			if err != nil {
				return err
			}
		}

		if decodedImage.Bounds().Max.Y < monitor.Height {
			y = monitor.Height - (((monitor.Height) - (decodedImage.Bounds().Max.Y)) / 2) - decodedImage.Bounds().Max.Y + monitor.PosY
		}

		if decodedImage.Bounds().Max.Y > monitor.Height {
			y = monitor.Height - (((decodedImage.Bounds().Max.Y) - (monitor.Height)) / 2) - monitor.Height + monitor.PosY
			canvasHeight = decodedImage.Bounds().Max.Y - (((decodedImage.Bounds().Max.Y) - (monitor.Height)) / 2)
			decodedImage, err = cutter.Crop(decodedImage, cutter.Config{
				Height: monitor.Height,
				Width:  decodedImage.Bounds().Max.X,
				Mode:   cutter.Centered,
			})
			if err != nil {
				return err
			}
		}
	}

	spMin := image.Point{X: x, Y: y}
	// max is bottom left points plus Width and Height
	spMax := image.Point{X: x + canvasWidth, Y: y + canvasHeight}
	// Create a rectangle the size of the monitor
	monitorRectangle := image.Rectangle{Min: spMin, Max: spMax}

	draw.Draw(drawImage, monitorRectangle, decodedImage, image.Point{X: 0, Y: 0}, draw.Over)

	decodedImage = nil

	return nil
}

Fig. 7. ProcessMonitorImage - Positioning and Saving

Fig. 8 shows visual representations and short descriptions for each image display mode that can be passed into processMonitorImage as an argument.

Fig. 8. Image Display Modes



Conclusion

Thanks very much for reading my article! My hope is that with this, and with future articles I can help people learn new stuff about programming. I'm by no means an expert, so bear in mind that the way I approach a problem may well not lead me to the perfect solution. Have a nice day!